[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:21:10 dist-test-slave-j0m4 krb5kdc[28970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:21:10 dist-test-slave-j0m4 krb5kdc[28970](info): set up 2 sockets
Sep 23 17:21:10 dist-test-slave-j0m4 krb5kdc[28970](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 17:21:11.724407 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:21:11 dist-test-slave-j0m4 krb5kdc[28970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:21:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:21:11.936583 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:36885
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:37179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:36885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:12.297195 28983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:12.297600 28983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:12.297766 28983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:12.305747 28983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:12.305943 28983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:12.306062 28983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:12.306167 28983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:12.306245 28983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:21:12.316807 28983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:37179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:36885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:36885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:12.318289 28983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:12.319774 28983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:12.334260 28988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:12.334385 28989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:12.334260 28991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:12.334897 28983 server_base.cc:1034] running on GCE node
I20260923 17:21:12.336177 28983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:12.338061 28983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:12.339326 28983 hybrid_clock.cc:648] HybridClock initialized: now 1790184072339272 us; error 43 us; skew 500 ppm
I20260923 17:21:12.339829 28983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:12 dist-test-slave-j0m4 krb5kdc[28970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184072, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:12.343623 28983 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:21:12.345480 28983 webserver.cc:460] Webserver started at http://127.28.72.254:34675/ using document root <none> and password file <none>
I20260923 17:21:12.346453 28983 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:12.346626 28983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:12.346994 28983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:12.350867 28983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "112ff69106ea415cb2a534b9dd9a2aee"
format_stamp: "Formatted at 2026-09-23 17:21:12 on dist-test-slave-j0m4"
server_key: "d3f18e6718e6fd6a54aee67b7b6ea21a"
server_key_iv: "baa94da35cf00dc7336c236f40b83c04"
server_key_version: "encryptionkey@0"
I20260923 17:21:12.351845 28983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "112ff69106ea415cb2a534b9dd9a2aee"
format_stamp: "Formatted at 2026-09-23 17:21:12 on dist-test-slave-j0m4"
server_key: "d3f18e6718e6fd6a54aee67b7b6ea21a"
server_key_iv: "baa94da35cf00dc7336c236f40b83c04"
server_key_version: "encryptionkey@0"
I20260923 17:21:12.357980 28983 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 17:21:12.362425 28998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:12.364220 28983 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:21:12.364473 28983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "112ff69106ea415cb2a534b9dd9a2aee"
format_stamp: "Formatted at 2026-09-23 17:21:12 on dist-test-slave-j0m4"
server_key: "d3f18e6718e6fd6a54aee67b7b6ea21a"
server_key_iv: "baa94da35cf00dc7336c236f40b83c04"
server_key_version: "encryptionkey@0"
I20260923 17:21:12.364717 28983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:12.392077 28983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:12.397301 28983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:12.397725 28983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:12.419059 28983 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:36885
I20260923 17:21:12.419102 29049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:36885 every 8 connection(s)
I20260923 17:21:12.420581 28983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:12.424944 29050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:12.427471 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 28983
I20260923 17:21:12.427786 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:21:12.429720 28963 external_mini_cluster.cc:1428] Setting key f9dba44d32ccd7407e84cc5151448830
I20260923 17:21:12.436053 29050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee: Bootstrap starting.
I20260923 17:21:12.439754 29050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:12.440815 29050 log.cc:824] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:12.443925 29050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee: No bootstrap required, opened a new log
I20260923 17:21:12.455369 29050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112ff69106ea415cb2a534b9dd9a2aee" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36885 } }
I20260923 17:21:12.455780 29050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:12.455850 29050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 112ff69106ea415cb2a534b9dd9a2aee, State: Initialized, Role: FOLLOWER
I20260923 17:21:12.456635 29050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112ff69106ea415cb2a534b9dd9a2aee" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36885 } }
I20260923 17:21:12.456959 29050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:12.457088 29050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:12.457284 29050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:12.460063 29050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112ff69106ea415cb2a534b9dd9a2aee" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36885 } }
I20260923 17:21:12.460683 29050 leader_election.cc:302] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 112ff69106ea415cb2a534b9dd9a2aee; no voters:
I20260923 17:21:12.461742 29050 leader_election.cc:288] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:12.461932 29055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:12.464119 29050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:12.464699 29055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [term 1 LEADER]: Becoming Leader. State: Replica: 112ff69106ea415cb2a534b9dd9a2aee, State: Running, Role: LEADER
Sep 23 17:21:12 dist-test-slave-j0m4 krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:12.465399 29055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112ff69106ea415cb2a534b9dd9a2aee" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36885 } }
I20260923 17:21:12.470660 29057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 112ff69106ea415cb2a534b9dd9a2aee. Latest consensus state: current_term: 1 leader_uuid: "112ff69106ea415cb2a534b9dd9a2aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112ff69106ea415cb2a534b9dd9a2aee" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36885 } } }
I20260923 17:21:12.471092 29057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:12.478803 29063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:12.478418 29056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "112ff69106ea415cb2a534b9dd9a2aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112ff69106ea415cb2a534b9dd9a2aee" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36885 } } }
I20260923 17:21:12.479894 29056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:12.483160 29052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:12.434064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.28.72.254:36885)
0923 17:21:12.435047 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:12.435063 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:12.436510 (+ 1447us) server_negotiation.cc:407] Connection header received
0923 17:21:12.446273 (+ 9763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:12.446306 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:12.446905 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:12.447440 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:12.451648 (+ 4208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:12.453967 (+ 2319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:12.456706 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:12.457952 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:12.458187 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:12.467394 (+ 9207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:12.467411 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:12.467442 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:12.467517 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:12.473344 (+ 5827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:12.475087 (+ 1743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:12.475101 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:12.475135 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:12.475443 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:12.476130 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:12.476134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:12.476137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:12.479188 (+ 3051us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:12.480139 (+ 951us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:12.480779 (+ 640us) server_negotiation.cc:299] Negotiation successful
0923 17:21:12.481233 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":452,"threads_started":1}
I20260923 17:21:12.487905 29063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:12.499341 29063 catalog_manager.cc:1347] Generated new cluster ID: 08c08fe53a3d4cd7a423adaf4d49a4cd
I20260923 17:21:12.499544 29063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:12.525588 29063 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:12.527676 29063 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:12.539939 29063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 112ff69106ea415cb2a534b9dd9a2aee: Generated new TSK 0
I20260923 17:21:12.540881 29063 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:21:12.599243 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:36885
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:37179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:12.970849 29077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:12.971246 29077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:12.971416 29077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:12.979943 29077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:12.980068 29077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:12.980266 29077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:21:12.990810 29077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:37179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:36885
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:12.992246 29077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:12.993640 29077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:13.008546 29083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:13.008682 29085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:13.008725 29082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:13.009321 29077 server_base.cc:1034] running on GCE node
I20260923 17:21:13.009976 29077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:13.010938 29077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:13.012161 29077 hybrid_clock.cc:648] HybridClock initialized: now 1790184073012126 us; error 65 us; skew 500 ppm
I20260923 17:21:13.012542 29077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:13 dist-test-slave-j0m4 krb5kdc[28970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:13.015947 29077 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:21:13.017732 29077 webserver.cc:460] Webserver started at http://127.28.72.193:36997/ using document root <none> and password file <none>
I20260923 17:21:13.018496 29077 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:13.018635 29077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:13.019016 29077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:13.022145 29077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "b62ec866a2be401b9347d08ac0e6d995"
format_stamp: "Formatted at 2026-09-23 17:21:13 on dist-test-slave-j0m4"
server_key: "e41e4288ba6323105159d36605af780c"
server_key_iv: "e9f49e3029a0e34454133312bd47ef22"
server_key_version: "encryptionkey@0"
I20260923 17:21:13.022998 29077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "b62ec866a2be401b9347d08ac0e6d995"
format_stamp: "Formatted at 2026-09-23 17:21:13 on dist-test-slave-j0m4"
server_key: "e41e4288ba6323105159d36605af780c"
server_key_iv: "e9f49e3029a0e34454133312bd47ef22"
server_key_version: "encryptionkey@0"
I20260923 17:21:13.028577 29077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260923 17:21:13.032773 29092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:13.034585 29077 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:21:13.034817 29077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "b62ec866a2be401b9347d08ac0e6d995"
format_stamp: "Formatted at 2026-09-23 17:21:13 on dist-test-slave-j0m4"
server_key: "e41e4288ba6323105159d36605af780c"
server_key_iv: "e9f49e3029a0e34454133312bd47ef22"
server_key_version: "encryptionkey@0"
I20260923 17:21:13.035045 29077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:13.052644 29077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:13.057825 29077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:13.058094 29077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:13.059209 29077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:13.060673 29077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:13.060752 29077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:13.060839 29077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:13.060889 29077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:13.098789 29077 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:37583
I20260923 17:21:13.098801 29204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:37583 every 8 connection(s)
I20260923 17:21:13.100273 29077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:21:13.109197 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 29077
I20260923 17:21:13.109444 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:21:13.110080 28963 external_mini_cluster.cc:1428] Setting key ce3468a29049093a7b73f94c2f855226
Sep 23 17:21:13 dist-test-slave-j0m4 krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:13.122390 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:13.103042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:50363 (local address 127.28.72.254:36885)
0923 17:21:13.103575 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:13.103580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:13.104411 (+ 831us) server_negotiation.cc:407] Connection header received
0923 17:21:13.106273 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:13.106279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:13.106374 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:13.106505 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:13.108190 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:13.108929 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:13.110911 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:13.111440 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:13.111561 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:13.115641 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:13.115646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:13.115650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:13.115684 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:13.119033 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:13.119939 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:13.119944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:13.119946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:13.120075 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:13.120565 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:13.120571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:13.120573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:13.120838 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:13.121000 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:13.121769 (+ 769us) server_negotiation.cc:299] Negotiation successful
0923 17:21:13.122017 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":294,"threads_started":1}
I20260923 17:21:13.123761 29208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:13.103543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36885 (local address 127.28.72.193:50363)
0923 17:21:13.104240 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:13.104284 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:13.105957 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:13.106775 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:13.106784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:13.107369 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:13.107989 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:13.108007 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:13.109099 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:13.109109 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:13.110626 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:13.110664 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:13.111700 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:13.111708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:13.112083 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:13.112909 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:13.112932 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:13.115369 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:13.119253 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:13.119266 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:13.119288 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:13.119778 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:13.120201 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:13.120206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:13.120210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:13.120422 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:13.120974 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:13.120991 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:13.121383 (+ 392us) client_negotiation.cc:769] Sending connection context
0923 17:21:13.121656 (+ 273us) client_negotiation.cc:240] Negotiation successful
0923 17:21:13.122247 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":262,"threads_started":1}
I20260923 17:21:13.126153 29205 heartbeater.cc:347] Connected to a master server at 127.28.72.254:36885
I20260923 17:21:13.126628 29205 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:13.127576 29205 heartbeater.cc:511] Master 127.28.72.254:36885 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
I20260923 17:21:13.130096 29015 ts_manager.cc:194] Registered new tserver with Master: b62ec866a2be401b9347d08ac0e6d995 (127.28.72.193:37583)
I20260923 17:21:13.131685 29015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:50363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:21:13.158901 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:36885
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:37179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:13.517561 29211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:13.517863 29211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:13.517963 29211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:13.525830 29211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:13.525950 29211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:13.526068 29211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:21:13.537370 29211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:37179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:36885
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:13.539538 29211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:13.541543 29211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:13.558172 29217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:13.558162 29219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:13.558622 29216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:13.559176 29211 server_base.cc:1034] running on GCE node
I20260923 17:21:13.559890 29211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:13.561174 29211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:13.562476 29211 hybrid_clock.cc:648] HybridClock initialized: now 1790184073562435 us; error 41 us; skew 500 ppm
I20260923 17:21:13.562880 29211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:13 dist-test-slave-j0m4 krb5kdc[28970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:13.567060 29211 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:21:13.569730 29211 webserver.cc:460] Webserver started at http://127.28.72.194:39713/ using document root <none> and password file <none>
I20260923 17:21:13.570591 29211 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:13.570719 29211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:13.571178 29211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:13.574874 29211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "57baa1ea0e0746399e959a88a6b436c2"
format_stamp: "Formatted at 2026-09-23 17:21:13 on dist-test-slave-j0m4"
server_key: "bdd1181978617af2d781afdb689d3882"
server_key_iv: "cef1942a49297a8ec22831930a9d44f1"
server_key_version: "encryptionkey@0"
I20260923 17:21:13.575877 29211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "57baa1ea0e0746399e959a88a6b436c2"
format_stamp: "Formatted at 2026-09-23 17:21:13 on dist-test-slave-j0m4"
server_key: "bdd1181978617af2d781afdb689d3882"
server_key_iv: "cef1942a49297a8ec22831930a9d44f1"
server_key_version: "encryptionkey@0"
I20260923 17:21:13.582614 29211 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 17:21:13.587414 29226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:13.589239 29211 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:21:13.589485 29211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "57baa1ea0e0746399e959a88a6b436c2"
format_stamp: "Formatted at 2026-09-23 17:21:13 on dist-test-slave-j0m4"
server_key: "bdd1181978617af2d781afdb689d3882"
server_key_iv: "cef1942a49297a8ec22831930a9d44f1"
server_key_version: "encryptionkey@0"
I20260923 17:21:13.589762 29211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:13.609027 29211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:13.614693 29211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:13.615036 29211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:13.616292 29211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:13.617904 29211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:13.617988 29211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:13.618079 29211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:13.618157 29211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:13.665136 29211 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:32787
I20260923 17:21:13.665155 29338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:32787 every 8 connection(s)
I20260923 17:21:13.666867 29211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:21:13.669153 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 29211
I20260923 17:21:13.669373 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:21:13.670011 28963 external_mini_cluster.cc:1428] Setting key 97fb3233524b50d8fdab85f142b712a8
Sep 23 17:21:13 dist-test-slave-j0m4 krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
I20260923 17:21:13.696255 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:13.670454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:51081 (local address 127.28.72.254:36885)
0923 17:21:13.671062 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:13.671069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:13.672631 (+ 1562us) server_negotiation.cc:407] Connection header received
0923 17:21:13.675411 (+ 2780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:13.675421 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:13.675540 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:13.675701 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:13.677863 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:13.679503 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:13.682473 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:13.683197 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:13.683384 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:13.688600 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:13.688606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:13.688609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:13.688657 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:13.691567 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:13.692715 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:13.692721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:13.692724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:13.692872 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:13.693629 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:13.693634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:13.693636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:13.694613 (+ 977us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:13.694760 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:13.695757 (+ 997us) server_negotiation.cc:299] Negotiation successful
0923 17:21:13.695970 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":292,"threads_started":1}
I20260923 17:21:13.697484 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:13.671361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36885 (local address 127.28.72.194:51081)
0923 17:21:13.672353 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:13.672409 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:13.674926 (+ 2517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:13.676047 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:13.676058 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:13.676758 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:13.677600 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:13.677631 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:13.679822 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:13.679829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:13.682209 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:13.682239 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:13.683574 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:13.683586 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:13.683937 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:13.684970 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:13.685003 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:13.688145 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:13.691782 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:13.691792 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:13.691810 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:13.692534 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:13.693065 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:13.693070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:13.693072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:13.693300 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:13.694803 (+ 1503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:13.694810 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:13.695338 (+ 528us) client_negotiation.cc:769] Sending connection context
0923 17:21:13.695670 (+ 332us) client_negotiation.cc:240] Negotiation successful
0923 17:21:13.696176 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":425,"threads_started":1}
I20260923 17:21:13.699903 29339 heartbeater.cc:347] Connected to a master server at 127.28.72.254:36885
I20260923 17:21:13.700322 29339 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:13.701593 29339 heartbeater.cc:511] Master 127.28.72.254:36885 requested a full tablet report, sending...
I20260923 17:21:13.703506 29015 ts_manager.cc:194] Registered new tserver with Master: 57baa1ea0e0746399e959a88a6b436c2 (127.28.72.194:32787)
I20260923 17:21:13.704438 29015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:51081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
I20260923 17:21:14.135438 29205 heartbeater.cc:503] Master 127.28.72.254:36885 was elected leader, sending a full tablet report...
I20260923 17:21:14.707691 29339 heartbeater.cc:503] Master 127.28.72.254:36885 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
W20260923 17:21:14.721459 28963 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.28.72.195: real 1.049s user 0.000s sys 0.008s
I20260923 17:21:14.722414 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:36885
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:37179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:15.111025 29345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:15.111425 29345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:15.111529 29345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:15.119663 29345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:15.119798 29345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:15.120002 29345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:21:15.130789 29345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:37179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:36885
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:15.132741 29345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:15.134205 29345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:15.150457 29353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:15.150481 29351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:15.150501 29350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:15.151851 29345 server_base.cc:1034] running on GCE node
I20260923 17:21:15.152684 29345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:15.153790 29345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:15.155097 29345 hybrid_clock.cc:648] HybridClock initialized: now 1790184075155048 us; error 49 us; skew 500 ppm
I20260923 17:21:15.155607 29345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:15 dist-test-slave-j0m4 krb5kdc[28970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184075, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:15.164817 29345 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:21:15.166535 29345 webserver.cc:460] Webserver started at http://127.28.72.195:37575/ using document root <none> and password file <none>
I20260923 17:21:15.167327 29345 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:15.167836 29345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:15.168134 29345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:15.171392 29345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "33c5157682a943f4bcc6895e6ea4f77d"
format_stamp: "Formatted at 2026-09-23 17:21:15 on dist-test-slave-j0m4"
server_key: "66717e506c56517be4d2e3125e7c108e"
server_key_iv: "0a0f7f6e8b01e3e4dc7dcc8af1ac3291"
server_key_version: "encryptionkey@0"
I20260923 17:21:15.172230 29345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "33c5157682a943f4bcc6895e6ea4f77d"
format_stamp: "Formatted at 2026-09-23 17:21:15 on dist-test-slave-j0m4"
server_key: "66717e506c56517be4d2e3125e7c108e"
server_key_iv: "0a0f7f6e8b01e3e4dc7dcc8af1ac3291"
server_key_version: "encryptionkey@0"
I20260923 17:21:15.178148 29345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:21:15.182436 29360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:15.184181 29345 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260923 17:21:15.184405 29345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "33c5157682a943f4bcc6895e6ea4f77d"
format_stamp: "Formatted at 2026-09-23 17:21:15 on dist-test-slave-j0m4"
server_key: "66717e506c56517be4d2e3125e7c108e"
server_key_iv: "0a0f7f6e8b01e3e4dc7dcc8af1ac3291"
server_key_version: "encryptionkey@0"
I20260923 17:21:15.184646 29345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:15.204000 29345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:15.208813 29345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:15.209131 29345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:15.210649 29345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:15.212208 29345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:15.212319 29345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:15.212471 29345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:15.212548 29345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:15.254802 29345 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:40629
I20260923 17:21:15.254848 29472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:40629 every 8 connection(s)
I20260923 17:21:15.256196 29345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:21:15.264051 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 29345
I20260923 17:21:15.264633 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:21:15.265159 28963 external_mini_cluster.cc:1428] Setting key 4c5b547a467c7b51cef8c93874563aa4
Sep 23 17:21:15 dist-test-slave-j0m4 krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184075, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:15.279336 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:15.258930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:52357 (local address 127.28.72.254:36885)
0923 17:21:15.259466 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:15.259473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:15.260307 (+ 834us) server_negotiation.cc:407] Connection header received
0923 17:21:15.262248 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:15.262254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:15.262347 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:15.262473 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:15.264911 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:15.265994 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:15.268010 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:15.268614 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:15.268754 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:15.272710 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:15.272716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:15.272719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:15.272754 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:15.275662 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:15.276622 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:15.276626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:15.276628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:15.276751 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:15.277290 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:15.277295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:15.277297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:15.277592 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:15.277797 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:15.278631 (+ 834us) server_negotiation.cc:299] Negotiation successful
0923 17:21:15.278901 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":504,"threads_started":1}
I20260923 17:21:15.280670 29476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:15.259433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36885 (local address 127.28.72.195:52357)
0923 17:21:15.260133 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:15.260178 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:15.261936 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:15.262729 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:15.262737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:15.263328 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:15.264701 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:15.264730 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:15.266186 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:15.266191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:15.267841 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:15.267854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:15.268960 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:15.268971 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:15.269213 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:15.270081 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:15.270122 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:15.272435 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:15.275836 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:15.275844 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:15.275860 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:15.276366 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:15.276873 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:15.276876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:15.276879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:15.277098 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:15.277811 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:15.277817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:15.278187 (+ 370us) client_negotiation.cc:769] Sending connection context
0923 17:21:15.278520 (+ 333us) client_negotiation.cc:240] Negotiation successful
0923 17:21:15.279106 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":261,"threads_started":1}
I20260923 17:21:15.282258 29473 heartbeater.cc:347] Connected to a master server at 127.28.72.254:36885
I20260923 17:21:15.282545 29473 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:15.283324 29473 heartbeater.cc:511] Master 127.28.72.254:36885 requested a full tablet report, sending...
I20260923 17:21:15.284945 29014 ts_manager.cc:194] Registered new tserver with Master: 33c5157682a943f4bcc6895e6ea4f77d (127.28.72.195:40629)
I20260923 17:21:15.285825 29014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:52357
I20260923 17:21:15.295248 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:21:15 dist-test-slave-j0m4 krb5kdc[28970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184075, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:21:15 dist-test-slave-j0m4 krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184075, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:21:15.345248 29481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:15.326988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36076 (local address 127.28.72.193:37583)
0923 17:21:15.327700 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:15.327707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:15.327735 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:21:15.327828 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:15.327836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:15.328052 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:15.328188 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:15.329635 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:15.330760 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:15.332231 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:15.332895 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:15.333077 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:15.337649 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:15.337664 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:15.337683 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:15.337724 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:15.341516 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:15.342633 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:15.342641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:15.342643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:15.342819 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:15.343388 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:15.343394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:15.343396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:15.343878 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:15.344186 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:15.344479 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 17:21:15.344948 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":360,"threads_started":1}
W20260923 17:21:15.347559 29120 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36076
Sep 23 17:21:15 dist-test-slave-j0m4 krb5kdc[28970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:21:15 dist-test-slave-j0m4 krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:21:15.442237 29481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:15.425410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.28.72.193:37583)
0923 17:21:15.425980 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:15.425989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:15.426017 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:21:15.426307 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:15.426311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:15.426379 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:15.426474 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:15.428438 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:15.429245 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:15.430632 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:15.431242 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:15.431351 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:15.435620 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:15.435625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:15.435629 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:15.435670 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:15.439188 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:15.440095 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:15.440100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:15.440110 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:15.440268 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:15.440799 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:15.440804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:15.440806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:15.441113 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:15.441328 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:15.441795 (+ 467us) server_negotiation.cc:299] Negotiation successful
0923 17:21:15.441997 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402}
I20260923 17:21:15.444200 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 29077
I20260923 17:21:15.459209 29481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:15.445551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36094 (local address 127.28.72.193:37583)
0923 17:21:15.445758 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:15.445763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:15.445888 (+ 125us) server_negotiation.cc:407] Connection header received
0923 17:21:15.446082 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:15.446087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:15.446148 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:15.446236 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:15.447607 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:15.448349 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:15.449708 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:15.450210 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:15.450368 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:15.452911 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:15.452917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:15.452921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:15.452956 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:15.456089 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:15.456926 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:15.456932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:15.456941 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:15.457116 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:15.457540 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:15.457547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:15.457553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:15.457845 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:15.458118 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:15.458420 (+ 302us) server_negotiation.cc:299] Negotiation successful
0923 17:21:15.458673 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:21:15.460248 29200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:15.554119 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 29077
I20260923 17:21:15.571825 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 29211
Sep 23 17:21:15 dist-test-slave-j0m4 krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:21:15.592505 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:15.573123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.28.72.194:32787)
0923 17:21:15.573999 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:15.574007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:15.574037 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:21:15.574104 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:15.574111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:15.574724 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:15.574897 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:15.576655 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:15.577856 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:15.579591 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:15.580241 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:15.580405 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:15.585256 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:15.585273 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:15.585291 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:15.585338 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:15.588653 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:15.589705 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:15.589713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:15.589716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:15.589854 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:15.590442 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:15.590447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:15.590449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:15.590855 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:15.591247 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:15.591786 (+ 539us) server_negotiation.cc:299] Negotiation successful
0923 17:21:15.592099 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":510,"threads_started":1}
I20260923 17:21:15.594161 29334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:15.682845 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 29211
I20260923 17:21:15.706714 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 29345
Sep 23 17:21:15 dist-test-slave-j0m4 krb5kdc[28970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:21:15.728618 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:15.707906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.28.72.195:40629)
0923 17:21:15.708737 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:15.708744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:15.708775 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:21:15.708841 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:15.708848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:15.709077 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:15.709218 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:15.710691 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:15.712051 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:15.713683 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:15.714323 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:15.714537 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:15.720802 (+ 6265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:15.720820 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:15.720838 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:15.720889 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:15.724378 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:15.725431 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:15.725441 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:15.725443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:15.725679 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:15.726434 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:15.726439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:15.726441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:15.726882 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:15.727313 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:15.727707 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 17:21:15.728038 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":533,"threads_started":1}
I20260923 17:21:15.730276 29468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:15.825938 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 29345
I20260923 17:21:15.842607 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 28983
I20260923 17:21:15.844039 29045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:15.961596 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 28983
2026-09-23T17:21:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5405 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:21:17 dist-test-slave-j0m4 krb5kdc[29495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:21:17 dist-test-slave-j0m4 krb5kdc[29495](info): set up 2 sockets
Sep 23 17:21:17 dist-test-slave-j0m4 krb5kdc[29495](info): commencing operation
krb5kdc: starting...
W20260923 17:21:18.465435 28963 mini_kdc.cc:121] Time spent starting KDC: real 2.455s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260923 17:21:18.671000 28963 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.205s user 0.000s sys 0.006s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260923 17:21:18.831828 28963 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.161s user 0.001s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260923 17:21:19.033926 28963 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.202s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/test-user.keytab.
W20260923 17:21:19.190069 28963 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.156s user 0.000s sys 0.004s
Sep 23 17:21:19 dist-test-slave-j0m4 krb5kdc[29495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:21:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:21:19.283836 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:40157
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:40157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:19.621942 29508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:19.622323 29508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:19.622511 29508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:19.630157 29508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:19.630368 29508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:19.630483 29508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:19.630589 29508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:19.630693 29508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:21:19.641157 29508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:40157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:40157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:19.642807 29508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:19.644502 29508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:19.657660 29513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:19.657660 29516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:19.657719 29514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:19.659056 29508 server_base.cc:1034] running on GCE node
I20260923 17:21:19.659987 29508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:19.661680 29508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:19.662992 29508 hybrid_clock.cc:648] HybridClock initialized: now 1790184079662890 us; error 63 us; skew 500 ppm
I20260923 17:21:19.663470 29508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:19 dist-test-slave-j0m4 krb5kdc[29495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184079, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:19.670655 29508 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:21:19.672905 29508 webserver.cc:460] Webserver started at http://127.28.72.254:44579/ using document root <none> and password file <none>
I20260923 17:21:19.673988 29508 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:19.674166 29508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:19.674628 29508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:19.679433 29508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "0ce6441b769746618ba6196cd1b5583f"
format_stamp: "Formatted at 2026-09-23 17:21:19 on dist-test-slave-j0m4"
server_key: "12dcf3fe62b9f59ae0380e067c4becff"
server_key_iv: "7c8f37cd7163555081e5ebf1d42c189c"
server_key_version: "encryptionkey@0"
I20260923 17:21:19.680371 29508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "0ce6441b769746618ba6196cd1b5583f"
format_stamp: "Formatted at 2026-09-23 17:21:19 on dist-test-slave-j0m4"
server_key: "12dcf3fe62b9f59ae0380e067c4becff"
server_key_iv: "7c8f37cd7163555081e5ebf1d42c189c"
server_key_version: "encryptionkey@0"
I20260923 17:21:19.688822 29508 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260923 17:21:19.693931 29523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:19.696070 29508 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 17:21:19.696291 29508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "0ce6441b769746618ba6196cd1b5583f"
format_stamp: "Formatted at 2026-09-23 17:21:19 on dist-test-slave-j0m4"
server_key: "12dcf3fe62b9f59ae0380e067c4becff"
server_key_iv: "7c8f37cd7163555081e5ebf1d42c189c"
server_key_version: "encryptionkey@0"
I20260923 17:21:19.696548 29508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:19.717254 29508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:19.722170 29508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:19.722491 29508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:19.742528 29508 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:40157
I20260923 17:21:19.742549 29574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:40157 every 8 connection(s)
I20260923 17:21:19.744210 29508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:19.748374 29575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:19.754199 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 29508
I20260923 17:21:19.754451 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:21:19.755228 28963 external_mini_cluster.cc:1428] Setting key 38f6d9d44893dfb0ca12242c5661c6d5
I20260923 17:21:19.760639 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f: Bootstrap starting.
I20260923 17:21:19.764694 29575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:19.765755 29575 log.cc:824] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:19.768879 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f: No bootstrap required, opened a new log
Sep 23 17:21:19 dist-test-slave-j0m4 krb5kdc[29495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:19.779704 29575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce6441b769746618ba6196cd1b5583f" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40157 } }
I20260923 17:21:19.780239 29575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:19.780378 29575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce6441b769746618ba6196cd1b5583f, State: Initialized, Role: FOLLOWER
I20260923 17:21:19.781126 29575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce6441b769746618ba6196cd1b5583f" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40157 } }
I20260923 17:21:19.781435 29575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:19.781577 29575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:19.781754 29575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:19.781863 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:19.757496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.28.72.254:40157)
0923 17:21:19.758428 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:19.758444 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:19.758495 (+ 51us) server_negotiation.cc:407] Connection header received
0923 17:21:19.760291 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:19.760317 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:19.760747 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:19.761197 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:19.763194 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:19.764701 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:19.766080 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:19.767278 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:19.767492 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:19.772021 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:19.772033 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:19.772057 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:19.772116 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:19.776586 (+ 4470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:19.777522 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:19.777534 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:19.777547 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:19.777790 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:19.778218 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:19.778223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:19.778226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:19.779045 (+ 819us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:19.779366 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:19.779901 (+ 535us) server_negotiation.cc:299] Negotiation successful
0923 17:21:19.780395 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":461,"threads_started":1}
I20260923 17:21:19.784333 29575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce6441b769746618ba6196cd1b5583f" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40157 } }
I20260923 17:21:19.784936 29575 leader_election.cc:302] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ce6441b769746618ba6196cd1b5583f; no voters:
I20260923 17:21:19.785671 29575 leader_election.cc:288] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:19.785841 29580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:19.786585 29580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [term 1 LEADER]: Becoming Leader. State: Replica: 0ce6441b769746618ba6196cd1b5583f, State: Running, Role: LEADER
I20260923 17:21:19.787204 29580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce6441b769746618ba6196cd1b5583f" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40157 } }
I20260923 17:21:19.787993 29575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:19.790277 29581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ce6441b769746618ba6196cd1b5583f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce6441b769746618ba6196cd1b5583f" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40157 } } }
I20260923 17:21:19.790405 29582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ce6441b769746618ba6196cd1b5583f. Latest consensus state: current_term: 1 leader_uuid: "0ce6441b769746618ba6196cd1b5583f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce6441b769746618ba6196cd1b5583f" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40157 } } }
I20260923 17:21:19.790693 29581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:19.790825 29582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:19.795544 29588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:19.801141 29588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:19.810616 29588 catalog_manager.cc:1347] Generated new cluster ID: c5d706eb94ff4d2a997fe9a23d1d0bbe
I20260923 17:21:19.810755 29588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:19.823900 29588 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:19.825294 29588 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:19.839684 29588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ce6441b769746618ba6196cd1b5583f: Generated new TSK 0
I20260923 17:21:19.840873 29588 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:21:19.891906 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40157
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:20.241894 29601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:20.242254 29601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:20.242436 29601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:20.250281 29601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:20.250444 29601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:20.250636 29601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:21:20.261109 29601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:20.262519 29601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:20.263983 29601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:20.278049 29606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:20.278117 29607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:20.278230 29609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:20.279238 29601 server_base.cc:1034] running on GCE node
I20260923 17:21:20.279865 29601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:20.280891 29601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:20.282107 29601 hybrid_clock.cc:648] HybridClock initialized: now 1790184080282060 us; error 41 us; skew 500 ppm
I20260923 17:21:20.282474 29601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:20 dist-test-slave-j0m4 krb5kdc[29495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184080, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:20.290109 29601 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:21:20.291890 29601 webserver.cc:460] Webserver started at http://127.28.72.193:46659/ using document root <none> and password file <none>
I20260923 17:21:20.292699 29601 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:20.293133 29601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:20.293592 29601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:20.296922 29601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "fe64537a75bf41f098cc386ca87fdda5"
format_stamp: "Formatted at 2026-09-23 17:21:20 on dist-test-slave-j0m4"
server_key: "89e4a02fdd4cffb01bba8b75e6bf32b0"
server_key_iv: "4f54a1c14f8958e3708a1fe901828eee"
server_key_version: "encryptionkey@0"
I20260923 17:21:20.297701 29601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "fe64537a75bf41f098cc386ca87fdda5"
format_stamp: "Formatted at 2026-09-23 17:21:20 on dist-test-slave-j0m4"
server_key: "89e4a02fdd4cffb01bba8b75e6bf32b0"
server_key_iv: "4f54a1c14f8958e3708a1fe901828eee"
server_key_version: "encryptionkey@0"
I20260923 17:21:20.303395 29601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260923 17:21:20.307061 29616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:20.308648 29601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:21:20.308837 29601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "fe64537a75bf41f098cc386ca87fdda5"
format_stamp: "Formatted at 2026-09-23 17:21:20 on dist-test-slave-j0m4"
server_key: "89e4a02fdd4cffb01bba8b75e6bf32b0"
server_key_iv: "4f54a1c14f8958e3708a1fe901828eee"
server_key_version: "encryptionkey@0"
I20260923 17:21:20.309093 29601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:20.329190 29601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:20.334581 29601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:20.334990 29601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:20.336277 29601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:20.338181 29601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:20.338294 29601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:20.338470 29601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:20.338546 29601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:20.376073 29601 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:46755
I20260923 17:21:20.376121 29728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:46755 every 8 connection(s)
I20260923 17:21:20.377432 29601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:21:20.382944 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 29601
I20260923 17:21:20.383193 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:21:20.384001 28963 external_mini_cluster.cc:1428] Setting key a3ce8a05f766d59a3190a15fcc95189a
Sep 23 17:21:20 dist-test-slave-j0m4 krb5kdc[29495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184080, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
I20260923 17:21:20.403795 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:20.380470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:39023 (local address 127.28.72.254:40157)
0923 17:21:20.380917 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:20.380921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:20.382272 (+ 1351us) server_negotiation.cc:407] Connection header received
0923 17:21:20.385054 (+ 2782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:20.385059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:20.385163 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:20.385291 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:20.387795 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:20.388669 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:20.391662 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:20.392397 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:20.392544 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:20.397123 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:20.397132 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:20.397140 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:20.397200 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:20.400485 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:20.401345 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:20.401351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:20.401353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:20.401469 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:20.401935 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:20.401939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:20.401941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:20.402200 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:20.402351 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:20.403288 (+ 937us) server_negotiation.cc:299] Negotiation successful
0923 17:21:20.403527 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":224,"threads_started":1}
I20260923 17:21:20.404764 29732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:20.381130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40157 (local address 127.28.72.193:39023)
0923 17:21:20.382082 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:20.382129 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:20.384710 (+ 2581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:20.385578 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:20.385588 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:20.386547 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:20.387568 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:20.387603 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:20.388839 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:20.388849 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:20.391474 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:20.391496 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:20.392697 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:20.392704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:20.393141 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:20.393942 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:20.393984 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:20.396825 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:20.400659 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:20.400669 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:20.400695 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:20.401194 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:20.401589 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:20.401594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:20.401597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:20.401808 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:20.402330 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:20.402338 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:20.402694 (+ 356us) client_negotiation.cc:769] Sending connection context
0923 17:21:20.402967 (+ 273us) client_negotiation.cc:240] Negotiation successful
0923 17:21:20.403469 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":491,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
I20260923 17:21:20.406574 29729 heartbeater.cc:347] Connected to a master server at 127.28.72.254:40157
I20260923 17:21:20.406934 29729 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:20.407864 29729 heartbeater.cc:511] Master 127.28.72.254:40157 requested a full tablet report, sending...
I20260923 17:21:20.410144 29540 ts_manager.cc:194] Registered new tserver with Master: fe64537a75bf41f098cc386ca87fdda5 (127.28.72.193:46755)
I20260923 17:21:20.412254 29540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:39023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:21:20.431344 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40157
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:20.768802 29735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:20.769143 29735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:20.769326 29735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:20.776978 29735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:20.777155 29735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:20.777351 29735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:21:20.787632 29735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:20.789142 29735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:20.790573 29735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:20.806671 29743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:20.806752 29740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:20.806725 29741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:20.807796 29735 server_base.cc:1034] running on GCE node
I20260923 17:21:20.808580 29735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:20.809759 29735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:20.810979 29735 hybrid_clock.cc:648] HybridClock initialized: now 1790184080810935 us; error 46 us; skew 500 ppm
I20260923 17:21:20.811342 29735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:20 dist-test-slave-j0m4 krb5kdc[29495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184080, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:20.814882 29735 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:21:20.816565 29735 webserver.cc:460] Webserver started at http://127.28.72.194:33623/ using document root <none> and password file <none>
I20260923 17:21:20.817301 29735 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:20.817452 29735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:20.817792 29735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:20.820930 29735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "528507f6522c42bcbe1ed0364f11280c"
format_stamp: "Formatted at 2026-09-23 17:21:20 on dist-test-slave-j0m4"
server_key: "8015c3fae0f612789fa2914ddfb1f2a0"
server_key_iv: "d32c9736aadc705230f39d63fcb28ef7"
server_key_version: "encryptionkey@0"
I20260923 17:21:20.821777 29735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "528507f6522c42bcbe1ed0364f11280c"
format_stamp: "Formatted at 2026-09-23 17:21:20 on dist-test-slave-j0m4"
server_key: "8015c3fae0f612789fa2914ddfb1f2a0"
server_key_iv: "d32c9736aadc705230f39d63fcb28ef7"
server_key_version: "encryptionkey@0"
I20260923 17:21:20.827502 29735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:21:20.831700 29750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:20.833349 29735 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:21:20.833601 29735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "528507f6522c42bcbe1ed0364f11280c"
format_stamp: "Formatted at 2026-09-23 17:21:20 on dist-test-slave-j0m4"
server_key: "8015c3fae0f612789fa2914ddfb1f2a0"
server_key_iv: "d32c9736aadc705230f39d63fcb28ef7"
server_key_version: "encryptionkey@0"
I20260923 17:21:20.833855 29735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:20.872613 29735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:20.877564 29735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:20.877909 29735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:20.879051 29735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:20.880728 29735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:20.880831 29735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:20.880970 29735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:20.881034 29735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:20.917131 29735 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:44109
I20260923 17:21:20.917210 29862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:44109 every 8 connection(s)
I20260923 17:21:20.918485 29735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:21:20.921151 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 29735
I20260923 17:21:20.921355 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:21:20.921832 28963 external_mini_cluster.cc:1428] Setting key aa3fe9d0cadc3852b588bb67f59bd88a
Sep 23 17:21:20 dist-test-slave-j0m4 krb5kdc[29495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184080, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
I20260923 17:21:20.943884 29865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:20.921327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:47489 (local address 127.28.72.254:40157)
0923 17:21:20.922313 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:20.922319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:20.923222 (+ 903us) server_negotiation.cc:407] Connection header received
0923 17:21:20.925283 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:20.925292 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:20.925392 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:20.925539 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:20.927572 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:20.928615 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:20.931715 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:20.932167 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:20.932287 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:20.937517 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:20.937524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:20.937527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:20.937567 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:20.940444 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:20.941325 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:20.941333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:20.941341 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:20.941482 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:20.941946 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:20.941956 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:20.941959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:20.942324 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:20.942484 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:20.943383 (+ 899us) server_negotiation.cc:299] Negotiation successful
0923 17:21:20.943615 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":737,"threads_started":1}
I20260923 17:21:20.945612 29866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:20.922075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40157 (local address 127.28.72.194:47489)
0923 17:21:20.923025 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:20.923074 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:20.924957 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:20.925834 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:20.925844 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:20.926543 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:20.927363 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:20.927387 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:20.928817 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:20.928826 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:20.931298 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:20.931323 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:20.932457 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:20.932467 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:20.932899 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:20.934018 (+ 1119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:20.934056 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:20.937175 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:20.940615 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:20.940626 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:20.940646 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:20.941136 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:20.941606 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:20.941611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:20.941615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:20.941816 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:20.942457 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:20.942466 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:20.942900 (+ 434us) client_negotiation.cc:769] Sending connection context
0923 17:21:20.943244 (+ 344us) client_negotiation.cc:240] Negotiation successful
0923 17:21:20.943837 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":375,"threads_started":1}
I20260923 17:21:20.947464 29863 heartbeater.cc:347] Connected to a master server at 127.28.72.254:40157
I20260923 17:21:20.947887 29863 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:20.948685 29863 heartbeater.cc:511] Master 127.28.72.254:40157 requested a full tablet report, sending...
I20260923 17:21:20.950311 29540 ts_manager.cc:194] Registered new tserver with Master: 528507f6522c42bcbe1ed0364f11280c (127.28.72.194:44109)
I20260923 17:21:20.951321 29540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:47489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:21:20.971953 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40157
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:21.310999 29869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:21.311380 29869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:21.311626 29869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:21.319188 29869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:21.319363 29869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:21.319556 29869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:21:21.330443 29869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:21.331873 29869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:21.333585 29869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:21.348934 29875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:21.348888 29874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:21.348973 29877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:21.349901 29869 server_base.cc:1034] running on GCE node
I20260923 17:21:21.350593 29869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:21.351826 29869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:21.353109 29869 hybrid_clock.cc:648] HybridClock initialized: now 1790184081353028 us; error 77 us; skew 500 ppm
I20260923 17:21:21.353539 29869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:21 dist-test-slave-j0m4 krb5kdc[29495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184081, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:21.357460 29869 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:21:21.359352 29869 webserver.cc:460] Webserver started at http://127.28.72.195:33355/ using document root <none> and password file <none>
I20260923 17:21:21.360111 29869 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:21.360267 29869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:21.360595 29869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:21.363893 29869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "0ab12e206b6e4f1c8c4171108597af0d"
format_stamp: "Formatted at 2026-09-23 17:21:21 on dist-test-slave-j0m4"
server_key: "f6677055736ed899db8a759225fd36e2"
server_key_iv: "283698bcddd015dd9d7912c401be0659"
server_key_version: "encryptionkey@0"
I20260923 17:21:21.364842 29869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "0ab12e206b6e4f1c8c4171108597af0d"
format_stamp: "Formatted at 2026-09-23 17:21:21 on dist-test-slave-j0m4"
server_key: "f6677055736ed899db8a759225fd36e2"
server_key_iv: "283698bcddd015dd9d7912c401be0659"
server_key_version: "encryptionkey@0"
I20260923 17:21:21.370836 29869 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260923 17:21:21.374902 29884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:21.376828 29869 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:21:21.377068 29869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "0ab12e206b6e4f1c8c4171108597af0d"
format_stamp: "Formatted at 2026-09-23 17:21:21 on dist-test-slave-j0m4"
server_key: "f6677055736ed899db8a759225fd36e2"
server_key_iv: "283698bcddd015dd9d7912c401be0659"
server_key_version: "encryptionkey@0"
I20260923 17:21:21.377314 29869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:21.401036 29869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:21.406121 29869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:21.406440 29869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:21.407555 29869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:21.409042 29869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:21.409154 29869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:21.409309 29869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:21.409384 29869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:21.416049 29729 heartbeater.cc:503] Master 127.28.72.254:40157 was elected leader, sending a full tablet report...
I20260923 17:21:21.445626 29869 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:37619
I20260923 17:21:21.445684 29996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:37619 every 8 connection(s)
I20260923 17:21:21.447019 29869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:21:21.451687 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 29869
I20260923 17:21:21.451887 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:21:21.452545 28963 external_mini_cluster.cc:1428] Setting key dc4d5a7f5944f2b3f1a05fb80fd71cc8
Sep 23 17:21:21 dist-test-slave-j0m4 krb5kdc[29495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184081, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:21.471882 29999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.449980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:52967 (local address 127.28.72.254:40157)
0923 17:21:21.450488 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:21.450493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:21.451580 (+ 1087us) server_negotiation.cc:407] Connection header received
0923 17:21:21.454005 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:21.454010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:21.454078 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:21.454224 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:21.456386 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.457293 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.459042 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.459550 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.459659 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.463630 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:21.463636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:21.463639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:21.463679 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:21.467183 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:21.468458 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:21.468465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:21.468469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:21.468679 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:21.469433 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:21.469438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:21.469440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:21.469833 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:21.470015 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:21.471328 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0923 17:21:21.471552 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":287,"threads_started":1}
I20260923 17:21:21.472946 30000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.450504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40157 (local address 127.28.72.195:52967)
0923 17:21:21.451350 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:21.451423 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:21.453638 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:21.454561 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:21.454571 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:21.455404 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:21.456139 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:21.456173 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.457513 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.457517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:21.458846 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:21.458860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.459810 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.459816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:21.460098 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.460979 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:21.461005 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:21.463358 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:21.467478 (+ 4120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:21.467492 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:21.467519 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:21.468167 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:21.468878 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:21.468884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:21.468887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:21.469185 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:21.470024 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:21.470034 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:21.470652 (+ 618us) client_negotiation.cc:769] Sending connection context
0923 17:21:21.471009 (+ 357us) client_negotiation.cc:240] Negotiation successful
0923 17:21:21.471618 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":288,"threads_started":1}
I20260923 17:21:21.474789 29997 heartbeater.cc:347] Connected to a master server at 127.28.72.254:40157
I20260923 17:21:21.475220 29997 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:21.476027 29997 heartbeater.cc:511] Master 127.28.72.254:40157 requested a full tablet report, sending...
I20260923 17:21:21.477903 29540 ts_manager.cc:194] Registered new tserver with Master: 0ab12e206b6e4f1c8c4171108597af0d (127.28.72.195:37619)
I20260923 17:21:21.478807 29540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:52967
I20260923 17:21:21.483235 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:21:21.506220 29999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.490689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36308 (local address 127.28.72.254:40157)
0923 17:21:21.490964 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:21.490975 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:21.491521 (+ 546us) server_negotiation.cc:407] Connection header received
0923 17:21:21.491670 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:21.491674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:21.491756 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:21.491847 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:21.494533 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.495451 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.496752 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.497185 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.497389 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.499924 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:21.499929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:21.499932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:21.499974 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:21.502941 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:21.503789 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:21.503794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:21.503798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:21.503950 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:21.504675 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:21.504679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:21.504681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:21.505055 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:21.505263 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:21.505646 (+ 383us) server_negotiation.cc:299] Negotiation successful
0923 17:21:21.505855 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260923 17:21:21.514482 29540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:21:21.521903 29540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:21:21.549528 30010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.535784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:44109 (local address 127.0.0.1:39366)
0923 17:21:21.537004 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:21.537043 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:21.537262 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:21.538120 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:21.538131 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:21.538171 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:21.538694 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:21.538710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.541661 (+ 2951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.541670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:21.545054 (+ 3384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:21.545080 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.547902 (+ 2822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.547912 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:21.548242 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.548335 (+ 93us) client_negotiation.cc:769] Sending connection context
0923 17:21:21.548510 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 17:21:21.548821 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":28,"thread_start_us":1843,"threads_started":1}
I20260923 17:21:21.551924 30012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.535863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.28.72.194:44109)
0923 17:21:21.536750 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:21.536760 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:21.537176 (+ 416us) server_negotiation.cc:407] Connection header received
0923 17:21:21.537539 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:21.537550 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:21.537845 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:21.538897 (+ 1052us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:21.539212 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.541424 (+ 2212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.545347 (+ 3923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.547570 (+ 2223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.550624 (+ 3054us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.550994 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:21.551097 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 17:21:21.551420 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":433,"threads_started":1}
I20260923 17:21:21.555773 30011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.535837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:37619 (local address 127.0.0.1:44996)
0923 17:21:21.537966 (+ 2129us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:21.537989 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:21.538096 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:21.542226 (+ 4130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:21.542234 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:21.542274 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:21.542903 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:21.542922 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.547457 (+ 4535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.547466 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:21.552145 (+ 4679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:21.552164 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.554317 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.554324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:21.554517 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.554556 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 17:21:21.554653 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 17:21:21.554795 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1994,"mutex_wait_us":1,"thread_start_us":820,"threads_started":1}
I20260923 17:21:21.556617 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.536325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.28.72.195:37619)
0923 17:21:21.539192 (+ 2867us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:21.539201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:21.539236 (+ 35us) server_negotiation.cc:407] Connection header received
0923 17:21:21.539338 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:21.539347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:21.539619 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:21.539874 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:21.543270 (+ 3396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.547196 (+ 3926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.552375 (+ 5179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.554061 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.554202 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.554482 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:21.556072 (+ 1590us) server_negotiation.cc:299] Negotiation successful
0923 17:21:21.556275 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2637,"thread_start_us":2458,"threads_started":1}
I20260923 17:21:21.557883 29798 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf70e1bc2b2489abc2b3f96ffec686e (DEFAULT_TABLE table=test-table [id=1bc438a0c412455abeee4975356509e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:21.559521 29798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf70e1bc2b2489abc2b3f96ffec686e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:21.560170 30010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.539248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:46755 (local address 127.0.0.1:51446)
0923 17:21:21.550184 (+ 10936us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:21.550213 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:21.550363 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:21.552883 (+ 2520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:21.552892 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:21.552940 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:21.553212 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:21.553226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.555070 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.555075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:21.557603 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:21.557623 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.559211 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.559217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:21.559408 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.559443 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:21:21.559700 (+ 257us) client_negotiation.cc:240] Negotiation successful
0923 17:21:21.559856 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10781,"thread_start_us":13930,"threads_started":1}
I20260923 17:21:21.560187 30015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.539483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.28.72.193:46755)
0923 17:21:21.551822 (+ 12339us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:21.551833 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:21.551864 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:21:21.552024 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:21.552035 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:21.552365 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:21.552850 (+ 485us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:21.553442 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.554864 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.557908 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.558990 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.559178 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.559432 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:21.559672 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 17:21:21.559856 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12136,"thread_start_us":13064,"threads_started":1}
I20260923 17:21:21.560839 29932 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf70e1bc2b2489abc2b3f96ffec686e (DEFAULT_TABLE table=test-table [id=1bc438a0c412455abeee4975356509e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:21.562480 29932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf70e1bc2b2489abc2b3f96ffec686e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:21.566852 29664 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf70e1bc2b2489abc2b3f96ffec686e (DEFAULT_TABLE table=test-table [id=1bc438a0c412455abeee4975356509e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:21.568418 29664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf70e1bc2b2489abc2b3f96ffec686e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:21.575774 30016 tablet_bootstrap.cc:492] T 9cf70e1bc2b2489abc2b3f96ffec686e P 528507f6522c42bcbe1ed0364f11280c: Bootstrap starting.
I20260923 17:21:21.580443 30017 tablet_bootstrap.cc:492] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d: Bootstrap starting.
I20260923 17:21:21.580695 30016 tablet_bootstrap.cc:654] T 9cf70e1bc2b2489abc2b3f96ffec686e P 528507f6522c42bcbe1ed0364f11280c: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:21.582279 30016 log.cc:824] T 9cf70e1bc2b2489abc2b3f96ffec686e P 528507f6522c42bcbe1ed0364f11280c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:21.584021 30017 tablet_bootstrap.cc:654] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:21.585256 30017 log.cc:824] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:21.585906 30016 tablet_bootstrap.cc:492] T 9cf70e1bc2b2489abc2b3f96ffec686e P 528507f6522c42bcbe1ed0364f11280c: No bootstrap required, opened a new log
I20260923 17:21:21.586054 30019 tablet_bootstrap.cc:492] T 9cf70e1bc2b2489abc2b3f96ffec686e P fe64537a75bf41f098cc386ca87fdda5: Bootstrap starting.
I20260923 17:21:21.586248 30016 ts_tablet_manager.cc:1397] T 9cf70e1bc2b2489abc2b3f96ffec686e P 528507f6522c42bcbe1ed0364f11280c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260923 17:21:21.589355 30017 tablet_bootstrap.cc:492] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d: No bootstrap required, opened a new log
I20260923 17:21:21.589730 30017 ts_tablet_manager.cc:1397] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 17:21:21.590248 30019 tablet_bootstrap.cc:654] T 9cf70e1bc2b2489abc2b3f96ffec686e P fe64537a75bf41f098cc386ca87fdda5: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:21.591647 30019 log.cc:824] T 9cf70e1bc2b2489abc2b3f96ffec686e P fe64537a75bf41f098cc386ca87fdda5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:21.595161 30019 tablet_bootstrap.cc:492] T 9cf70e1bc2b2489abc2b3f96ffec686e P fe64537a75bf41f098cc386ca87fdda5: No bootstrap required, opened a new log
I20260923 17:21:21.595468 30019 ts_tablet_manager.cc:1397] T 9cf70e1bc2b2489abc2b3f96ffec686e P fe64537a75bf41f098cc386ca87fdda5: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260923 17:21:21.596457 30016 raft_consensus.cc:348] T 9cf70e1bc2b2489abc2b3f96ffec686e P 528507f6522c42bcbe1ed0364f11280c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528507f6522c42bcbe1ed0364f11280c" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44109 } } peers { permanent_uuid: "0ab12e206b6e4f1c8c4171108597af0d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37619 } } peers { permanent_uuid: "fe64537a75bf41f098cc386ca87fdda5" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46755 } }
I20260923 17:21:21.596880 30016 raft_consensus.cc:374] T 9cf70e1bc2b2489abc2b3f96ffec686e P 528507f6522c42bcbe1ed0364f11280c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:21.596990 30016 raft_consensus.cc:729] T 9cf70e1bc2b2489abc2b3f96ffec686e P 528507f6522c42bcbe1ed0364f11280c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528507f6522c42bcbe1ed0364f11280c, State: Initialized, Role: FOLLOWER
I20260923 17:21:21.597685 30016 consensus_queue.cc:260] T 9cf70e1bc2b2489abc2b3f96ffec686e P 528507f6522c42bcbe1ed0364f11280c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528507f6522c42bcbe1ed0364f11280c" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44109 } } peers { permanent_uuid: "0ab12e206b6e4f1c8c4171108597af0d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37619 } } peers { permanent_uuid: "fe64537a75bf41f098cc386ca87fdda5" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46755 } }
I20260923 17:21:21.598923 29863 heartbeater.cc:503] Master 127.28.72.254:40157 was elected leader, sending a full tablet report...
I20260923 17:21:21.599748 30016 ts_tablet_manager.cc:1428] T 9cf70e1bc2b2489abc2b3f96ffec686e P 528507f6522c42bcbe1ed0364f11280c: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260923 17:21:21.600432 30017 raft_consensus.cc:348] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528507f6522c42bcbe1ed0364f11280c" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44109 } } peers { permanent_uuid: "0ab12e206b6e4f1c8c4171108597af0d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37619 } } peers { permanent_uuid: "fe64537a75bf41f098cc386ca87fdda5" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46755 } }
I20260923 17:21:21.600883 30017 raft_consensus.cc:374] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:21.600979 30017 raft_consensus.cc:729] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab12e206b6e4f1c8c4171108597af0d, State: Initialized, Role: FOLLOWER
I20260923 17:21:21.601635 30017 consensus_queue.cc:260] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528507f6522c42bcbe1ed0364f11280c" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44109 } } peers { permanent_uuid: "0ab12e206b6e4f1c8c4171108597af0d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37619 } } peers { permanent_uuid: "fe64537a75bf41f098cc386ca87fdda5" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46755 } }
I20260923 17:21:21.603560 30017 ts_tablet_manager.cc:1428] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260923 17:21:21.604131 29997 heartbeater.cc:503] Master 127.28.72.254:40157 was elected leader, sending a full tablet report...
I20260923 17:21:21.606928 30019 raft_consensus.cc:348] T 9cf70e1bc2b2489abc2b3f96ffec686e P fe64537a75bf41f098cc386ca87fdda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528507f6522c42bcbe1ed0364f11280c" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44109 } } peers { permanent_uuid: "0ab12e206b6e4f1c8c4171108597af0d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37619 } } peers { permanent_uuid: "fe64537a75bf41f098cc386ca87fdda5" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46755 } }
I20260923 17:21:21.607645 30019 raft_consensus.cc:374] T 9cf70e1bc2b2489abc2b3f96ffec686e P fe64537a75bf41f098cc386ca87fdda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:21.607771 30019 raft_consensus.cc:729] T 9cf70e1bc2b2489abc2b3f96ffec686e P fe64537a75bf41f098cc386ca87fdda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe64537a75bf41f098cc386ca87fdda5, State: Initialized, Role: FOLLOWER
I20260923 17:21:21.608492 30019 consensus_queue.cc:260] T 9cf70e1bc2b2489abc2b3f96ffec686e P fe64537a75bf41f098cc386ca87fdda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528507f6522c42bcbe1ed0364f11280c" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44109 } } peers { permanent_uuid: "0ab12e206b6e4f1c8c4171108597af0d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37619 } } peers { permanent_uuid: "fe64537a75bf41f098cc386ca87fdda5" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46755 } }
I20260923 17:21:21.610287 30019 ts_tablet_manager.cc:1428] T 9cf70e1bc2b2489abc2b3f96ffec686e P fe64537a75bf41f098cc386ca87fdda5: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
W20260923 17:21:21.630028 29730 tablet.cc:2369] T 9cf70e1bc2b2489abc2b3f96ffec686e P fe64537a75bf41f098cc386ca87fdda5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:21:21.670866 29864 tablet.cc:2369] T 9cf70e1bc2b2489abc2b3f96ffec686e P 528507f6522c42bcbe1ed0364f11280c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:21:21.698810 29998 tablet.cc:2369] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:21:21.756333 30023 raft_consensus.cc:482] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:21:21.756676 30023 raft_consensus.cc:504] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528507f6522c42bcbe1ed0364f11280c" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44109 } } peers { permanent_uuid: "0ab12e206b6e4f1c8c4171108597af0d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37619 } } peers { permanent_uuid: "fe64537a75bf41f098cc386ca87fdda5" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46755 } }
I20260923 17:21:21.758925 30023 leader_election.cc:288] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 528507f6522c42bcbe1ed0364f11280c (127.28.72.194:44109), fe64537a75bf41f098cc386ca87fdda5 (127.28.72.193:46755)
I20260923 17:21:21.766219 30012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.759498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:36179 (local address 127.28.72.194:44109)
0923 17:21:21.759749 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:21.759755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:21.759943 (+ 188us) server_negotiation.cc:407] Connection header received
0923 17:21:21.760188 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:21.760194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:21.760282 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:21.760401 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:21.761002 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.762037 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.764007 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.765348 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.765460 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.765512 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:21.765899 (+ 387us) server_negotiation.cc:299] Negotiation successful
0923 17:21:21.766025 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:21:21.766302 30000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.759546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:44109 (local address 127.28.72.195:36179)
0923 17:21:21.759763 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:21.759785 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:21.759984 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:21.760610 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:21.760616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:21.760654 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:21.760844 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:21.760853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.762170 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.762177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:21.763837 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:21.763849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.765494 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.765501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:21.765695 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.765734 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 17:21:21.765833 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 17:21:21.765984 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 17:21:21.767660 30015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.760280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:41839 (local address 127.28.72.193:46755)
0923 17:21:21.760604 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:21.760611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:21.761010 (+ 399us) server_negotiation.cc:407] Connection header received
0923 17:21:21.761193 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:21.761198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:21.761257 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:21.761338 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:21.761943 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.763063 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.765290 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.766575 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.766739 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.766826 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:21.767380 (+ 554us) server_negotiation.cc:299] Negotiation successful
0923 17:21:21.767486 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260923 17:21:21.767679 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.759990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:46755 (local address 127.28.72.195:41839)
0923 17:21:21.760874 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:21.760892 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:21.761030 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:21.761390 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:21.761394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:21.761421 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:21.761767 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:21.761777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.763250 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.763253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:21.765121 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:21.765138 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.766827 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.766834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:21.767034 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.767068 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:21:21.767187 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 17:21:21.767329 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":522,"threads_started":1}
I20260923 17:21:21.767642 29818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf70e1bc2b2489abc2b3f96ffec686e" candidate_uuid: "0ab12e206b6e4f1c8c4171108597af0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528507f6522c42bcbe1ed0364f11280c" is_pre_election: true
I20260923 17:21:21.768277 29818 raft_consensus.cc:2454] T 9cf70e1bc2b2489abc2b3f96ffec686e P 528507f6522c42bcbe1ed0364f11280c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ab12e206b6e4f1c8c4171108597af0d in term 0.
I20260923 17:21:21.768884 29684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf70e1bc2b2489abc2b3f96ffec686e" candidate_uuid: "0ab12e206b6e4f1c8c4171108597af0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe64537a75bf41f098cc386ca87fdda5" is_pre_election: true
I20260923 17:21:21.769011 29887 leader_election.cc:302] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab12e206b6e4f1c8c4171108597af0d, 528507f6522c42bcbe1ed0364f11280c; no voters:
I20260923 17:21:21.769443 30023 raft_consensus.cc:2781] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:21:21.769415 29684 raft_consensus.cc:2454] T 9cf70e1bc2b2489abc2b3f96ffec686e P fe64537a75bf41f098cc386ca87fdda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ab12e206b6e4f1c8c4171108597af0d in term 0.
I20260923 17:21:21.769603 30023 raft_consensus.cc:482] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:21:21.769714 30023 raft_consensus.cc:3037] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:21.772457 30023 raft_consensus.cc:504] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528507f6522c42bcbe1ed0364f11280c" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44109 } } peers { permanent_uuid: "0ab12e206b6e4f1c8c4171108597af0d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37619 } } peers { permanent_uuid: "fe64537a75bf41f098cc386ca87fdda5" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46755 } }
I20260923 17:21:21.773244 30023 leader_election.cc:288] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [CANDIDATE]: Term 1 election: Requested vote from peers 528507f6522c42bcbe1ed0364f11280c (127.28.72.194:44109), fe64537a75bf41f098cc386ca87fdda5 (127.28.72.193:46755)
I20260923 17:21:21.774001 29818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf70e1bc2b2489abc2b3f96ffec686e" candidate_uuid: "0ab12e206b6e4f1c8c4171108597af0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528507f6522c42bcbe1ed0364f11280c"
I20260923 17:21:21.774000 29684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf70e1bc2b2489abc2b3f96ffec686e" candidate_uuid: "0ab12e206b6e4f1c8c4171108597af0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe64537a75bf41f098cc386ca87fdda5"
I20260923 17:21:21.774237 29684 raft_consensus.cc:3037] T 9cf70e1bc2b2489abc2b3f96ffec686e P fe64537a75bf41f098cc386ca87fdda5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:21.774237 29818 raft_consensus.cc:3037] T 9cf70e1bc2b2489abc2b3f96ffec686e P 528507f6522c42bcbe1ed0364f11280c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:21.777678 29818 raft_consensus.cc:2454] T 9cf70e1bc2b2489abc2b3f96ffec686e P 528507f6522c42bcbe1ed0364f11280c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ab12e206b6e4f1c8c4171108597af0d in term 1.
I20260923 17:21:21.777678 29684 raft_consensus.cc:2454] T 9cf70e1bc2b2489abc2b3f96ffec686e P fe64537a75bf41f098cc386ca87fdda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ab12e206b6e4f1c8c4171108597af0d in term 1.
I20260923 17:21:21.778272 29887 leader_election.cc:302] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab12e206b6e4f1c8c4171108597af0d, 528507f6522c42bcbe1ed0364f11280c; no voters:
I20260923 17:21:21.778678 30023 raft_consensus.cc:2781] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:21.779261 30023 raft_consensus.cc:686] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [term 1 LEADER]: Becoming Leader. State: Replica: 0ab12e206b6e4f1c8c4171108597af0d, State: Running, Role: LEADER
I20260923 17:21:21.779783 30023 consensus_queue.cc:237] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528507f6522c42bcbe1ed0364f11280c" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44109 } } peers { permanent_uuid: "0ab12e206b6e4f1c8c4171108597af0d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37619 } } peers { permanent_uuid: "fe64537a75bf41f098cc386ca87fdda5" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46755 } }
I20260923 17:21:21.785157 29540 catalog_manager.cc:5495] T 9cf70e1bc2b2489abc2b3f96ffec686e P 0ab12e206b6e4f1c8c4171108597af0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ab12e206b6e4f1c8c4171108597af0d (127.28.72.195). New cstate: current_term: 1 leader_uuid: "0ab12e206b6e4f1c8c4171108597af0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528507f6522c42bcbe1ed0364f11280c" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ab12e206b6e4f1c8c4171108597af0d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe64537a75bf41f098cc386ca87fdda5" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46755 } health_report { overall_health: UNKNOWN } } }
Sep 23 17:21:21 dist-test-slave-j0m4 krb5kdc[29495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:21:21.856706 30015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.839363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.28.72.193:46755)
0923 17:21:21.839601 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:21.839610 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:21.840085 (+ 475us) server_negotiation.cc:407] Connection header received
0923 17:21:21.840306 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:21.840311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:21.840388 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:21.840496 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:21.842441 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.843282 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.844601 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.845039 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.845180 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.849565 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:21.849584 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:21.849601 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:21.849642 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:21.853439 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:21.854310 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:21.854319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:21.854321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:21.854466 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:21.854933 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:21.854937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:21.854940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:21.855443 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:21.855676 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:21.856041 (+ 365us) server_negotiation.cc:299] Negotiation successful
0923 17:21:21.856216 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Sep 23 17:21:21 dist-test-slave-j0m4 krb5kdc[29495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184081, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:21:21 dist-test-slave-j0m4 krb5kdc[29495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184081, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:21:21.911938 30015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.894836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51474 (local address 127.28.72.193:46755)
0923 17:21:21.895063 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:21.895070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:21.896328 (+ 1258us) server_negotiation.cc:407] Connection header received
0923 17:21:21.896550 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:21.896557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:21.896669 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:21.896796 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:21.898597 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.899506 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.900800 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.901228 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.901383 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.906007 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:21.906013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:21.906017 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:21.906062 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:21.908990 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:21.909846 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:21.909859 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:21.909862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:21.910014 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:21.910436 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:21.910441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:21.910447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:21.910745 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:21.911057 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:21.911341 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 17:21:21.911592 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 17:21:21 dist-test-slave-j0m4 krb5kdc[29495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:21:21 dist-test-slave-j0m4 krb5kdc[29495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:21:21.963595 30015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.948025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.28.72.193:46755)
0923 17:21:21.948306 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:21.948316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:21.948747 (+ 431us) server_negotiation.cc:407] Connection header received
0923 17:21:21.948996 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:21.949006 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:21.949073 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:21.949182 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:21.950583 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.951368 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.952665 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.953113 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.953252 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.957524 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:21.957529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:21.957533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:21.957582 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:21.960684 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:21.961466 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:21.961473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:21.961535 (+ 62us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:21.961672 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:21.962072 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:21.962078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:21.962082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:21.962393 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:21.962661 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:21.962945 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 17:21:21.963230 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260923 17:21:21.965358 29644 tablet_service.cc:2908] Rejecting scan request for tablet 9cf70e1bc2b2489abc2b3f96ffec686e: Uninitialized: clean time has not yet been initialized
I20260923 17:21:21.968842 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 29601
I20260923 17:21:21.983991 30015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:21.970403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.28.72.193:46755)
0923 17:21:21.970624 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:21.970631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:21.970855 (+ 224us) server_negotiation.cc:407] Connection header received
0923 17:21:21.971019 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:21.971023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:21.971095 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:21.971226 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:21.972379 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.973075 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.974798 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:21.975471 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:21.975599 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:21.978011 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:21.978017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:21.978020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:21.978071 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:21.980883 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:21.981735 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:21.981741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:21.981744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:21.981933 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:21.982355 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:21.982359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:21.982361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:21.982659 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:21.982886 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:21.983339 (+ 453us) server_negotiation.cc:299] Negotiation successful
0923 17:21:21.983561 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:21:21.984781 29724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:22.078118 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 29601
I20260923 17:21:22.096633 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 29735
Sep 23 17:21:22 dist-test-slave-j0m4 krb5kdc[29495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:21:22.112560 30012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:22.097597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.28.72.194:44109)
0923 17:21:22.097817 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:22.097824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:22.097850 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:21:22.097903 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:22.097907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:22.097977 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:22.098094 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:22.099306 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:22.100032 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:22.101357 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:22.101760 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:22.101903 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:22.106461 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:22.106477 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:22.106493 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:22.106538 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:22.109439 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:22.110230 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:22.110238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:22.110240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:22.110422 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:22.110837 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:22.110842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:22.110845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:22.111321 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:22.111631 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:22.111954 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 17:21:22.112192 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:21:22.113260 29858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:22.202989 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 29735
I20260923 17:21:22.219835 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 29869
Sep 23 17:21:22 dist-test-slave-j0m4 krb5kdc[29495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:21:22.235450 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:22.220631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45000 (local address 127.28.72.195:37619)
0923 17:21:22.221108 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:22.221113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:22.221132 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:21:22.221189 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:22.221194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:22.221262 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:22.221362 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:22.222592 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:22.223408 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:22.224665 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:22.225114 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:22.225217 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:22.229441 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:22.229457 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:22.229473 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:22.229511 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:22.232408 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:22.233111 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:22.233119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:22.233136 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:22.233287 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:22.233719 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:22.233724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:22.233726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:22.234155 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:22.234414 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:22.234687 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 17:21:22.234880 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":219,"threads_started":1}
I20260923 17:21:22.236109 29992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:22.326335 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 29869
I20260923 17:21:22.343075 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 29508
I20260923 17:21:22.344465 29570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:22.453043 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 29508
2026-09-23T17:21:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (6489 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:21:22 dist-test-slave-j0m4 krb5kdc[30046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:21:22 dist-test-slave-j0m4 krb5kdc[30046](info): set up 2 sockets
Sep 23 17:21:22 dist-test-slave-j0m4 krb5kdc[30046](info): commencing operation
krb5kdc: starting...
W20260923 17:21:23.547467 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:21:23 dist-test-slave-j0m4 krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:21:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:21:23.752287 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:45335
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:45335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:24.103737 30059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:24.104110 30059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:24.104206 30059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:24.111792 30059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:24.111918 30059 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:21:24.111999 30059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:24.112100 30059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:24.112169 30059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:24.112255 30059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:21:24.122867 30059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:45335
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:45335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:24.124342 30059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:24.125857 30059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:24.139092 30065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:24.139171 30064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:24.139216 30067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:24.140285 30059 server_base.cc:1034] running on GCE node
I20260923 17:21:24.141245 30059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:24.142767 30059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:24.144001 30059 hybrid_clock.cc:648] HybridClock initialized: now 1790184084143924 us; error 50 us; skew 500 ppm
I20260923 17:21:24.144322 30059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:24 dist-test-slave-j0m4 krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184084, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:24.148033 30059 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:21:24.149744 30059 webserver.cc:460] Webserver started at http://127.28.72.254:36155/ using document root <none> and password file <none>
I20260923 17:21:24.150540 30059 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:24.150640 30059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:24.151001 30059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:24.154147 30059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "81b17736261949febc2c8ad1385b5b38"
format_stamp: "Formatted at 2026-09-23 17:21:24 on dist-test-slave-j0m4"
server_key: "c7601548fc6f1ea91b8561cae76f8112"
server_key_iv: "d219f66112aad7fb3c7105cff13a56c6"
server_key_version: "encryptionkey@0"
I20260923 17:21:24.155339 30059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "81b17736261949febc2c8ad1385b5b38"
format_stamp: "Formatted at 2026-09-23 17:21:24 on dist-test-slave-j0m4"
server_key: "c7601548fc6f1ea91b8561cae76f8112"
server_key_iv: "d219f66112aad7fb3c7105cff13a56c6"
server_key_version: "encryptionkey@0"
I20260923 17:21:24.161247 30059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:21:24.165382 30074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:24.166877 30059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:21:24.167098 30059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "81b17736261949febc2c8ad1385b5b38"
format_stamp: "Formatted at 2026-09-23 17:21:24 on dist-test-slave-j0m4"
server_key: "c7601548fc6f1ea91b8561cae76f8112"
server_key_iv: "d219f66112aad7fb3c7105cff13a56c6"
server_key_version: "encryptionkey@0"
I20260923 17:21:24.167372 30059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:24.188704 30059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:24.195178 30059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:24.195525 30059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:24.222560 30059 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:45335
I20260923 17:21:24.222524 30135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:45335 every 8 connection(s)
I20260923 17:21:24.224721 30059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:24.229300 30136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:24.233021 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 30059
I20260923 17:21:24.233284 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:21:24.234288 28963 external_mini_cluster.cc:1428] Setting key ed4a3f62d645348331af4be0cd45ab38
I20260923 17:21:24.245712 30136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38: Bootstrap starting.
I20260923 17:21:24.249806 30136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:24.251282 30136 log.cc:824] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:21:24 dist-test-slave-j0m4 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:24.255029 30136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38: No bootstrap required, opened a new log
I20260923 17:21:24.266633 30136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b17736261949febc2c8ad1385b5b38" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45335 } }
I20260923 17:21:24.267252 30136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:24.267534 30136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b17736261949febc2c8ad1385b5b38, State: Initialized, Role: FOLLOWER
I20260923 17:21:24.267606 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:24.236735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51258 (local address 127.28.72.254:45335)
0923 17:21:24.237730 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:24.237750 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:24.237809 (+ 59us) server_negotiation.cc:407] Connection header received
0923 17:21:24.241092 (+ 3283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:24.241145 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:24.241848 (+ 703us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:24.242419 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:24.244355 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:24.246912 (+ 2557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:24.249006 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:24.250491 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:24.250740 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:24.256177 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:24.256189 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:24.256210 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:24.256260 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:24.260681 (+ 4421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:24.261745 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:24.261756 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:24.261767 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:24.262022 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:24.262546 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:24.262552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:24.262555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:24.263460 (+ 905us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:24.263756 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:24.264560 (+ 804us) server_negotiation.cc:299] Negotiation successful
0923 17:21:24.265863 (+ 1303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":409,"threads_started":1}
I20260923 17:21:24.268491 30136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b17736261949febc2c8ad1385b5b38" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45335 } }
I20260923 17:21:24.268859 30136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:24.269032 30136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:24.269273 30136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:24.272948 30136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b17736261949febc2c8ad1385b5b38" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45335 } }
I20260923 17:21:24.273636 30136 leader_election.cc:302] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81b17736261949febc2c8ad1385b5b38; no voters:
I20260923 17:21:24.274312 30136 leader_election.cc:288] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:24.274551 30141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:24.275421 30141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [term 1 LEADER]: Becoming Leader. State: Replica: 81b17736261949febc2c8ad1385b5b38, State: Running, Role: LEADER
I20260923 17:21:24.276178 30141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b17736261949febc2c8ad1385b5b38" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45335 } }
I20260923 17:21:24.276986 30136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:24.280402 30142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81b17736261949febc2c8ad1385b5b38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b17736261949febc2c8ad1385b5b38" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45335 } } }
I20260923 17:21:24.280853 30142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:24.283928 30146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:24.283746 30143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81b17736261949febc2c8ad1385b5b38. Latest consensus state: current_term: 1 leader_uuid: "81b17736261949febc2c8ad1385b5b38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b17736261949febc2c8ad1385b5b38" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45335 } } }
I20260923 17:21:24.284303 30143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38 [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:24.290607 30146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:24.304453 30146 catalog_manager.cc:1347] Generated new cluster ID: 73083552f66c40028e94bd5a78a3dfe5
I20260923 17:21:24.304603 30146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:24.332684 30146 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:24.334357 30146 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:24.343700 30146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38: Generated new TSK 0
I20260923 17:21:24.345048 30146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:21:24.395817 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:45335
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:24.751312 30162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:24.751647 30162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:24.751827 30162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:24.760118 30162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:24.760252 30162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:24.760409 30162 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:21:24.760543 30162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:21:24.771585 30162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.72.254:45335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:24.773082 30162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:24.774552 30162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:24.789109 30170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:24.789057 30167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:24.789057 30168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:24.790655 30162 server_base.cc:1034] running on GCE node
I20260923 17:21:24.791491 30162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:24.792565 30162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:24.793898 30162 hybrid_clock.cc:648] HybridClock initialized: now 1790184084793782 us; error 52 us; skew 500 ppm
I20260923 17:21:24.794297 30162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:24 dist-test-slave-j0m4 krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184084, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:24.798158 30162 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:21:24.800112 30162 webserver.cc:460] Webserver started at http://127.28.72.193:44011/ using document root <none> and password file <none>
I20260923 17:21:24.800967 30162 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:24.801117 30162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:24.801472 30162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:24.804701 30162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8"
format_stamp: "Formatted at 2026-09-23 17:21:24 on dist-test-slave-j0m4"
server_key: "6fdccb9dbcd01c53777258798470d12e"
server_key_iv: "d1b132f46a957b01a7e5e2c77e94fd49"
server_key_version: "encryptionkey@0"
I20260923 17:21:24.805644 30162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8"
format_stamp: "Formatted at 2026-09-23 17:21:24 on dist-test-slave-j0m4"
server_key: "6fdccb9dbcd01c53777258798470d12e"
server_key_iv: "d1b132f46a957b01a7e5e2c77e94fd49"
server_key_version: "encryptionkey@0"
I20260923 17:21:24.811900 30162 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 17:21:24.816210 30177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:24.818427 30162 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:21:24.818673 30162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8"
format_stamp: "Formatted at 2026-09-23 17:21:24 on dist-test-slave-j0m4"
server_key: "6fdccb9dbcd01c53777258798470d12e"
server_key_iv: "d1b132f46a957b01a7e5e2c77e94fd49"
server_key_version: "encryptionkey@0"
I20260923 17:21:24.819020 30162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:24.840127 30162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:24.845721 30162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:24.846047 30162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:24.849144 30162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:24.849283 30162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:24.849444 30162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:24.849514 30162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:21:24 dist-test-slave-j0m4 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184084, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:24.880740 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:24.850814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:45335 (local address 127.28.72.193:47187)
0923 17:21:24.851865 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:24.851932 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:24.854692 (+ 2760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:24.855718 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:24.855756 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:24.856718 (+ 962us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:24.857700 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:24.857723 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:24.859826 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:24.859833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:24.862013 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:24.862030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:24.863314 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:24.863326 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:24.863780 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:24.864993 (+ 1213us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:24.865043 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:24.869342 (+ 4299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:24.874006 (+ 4664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:24.874023 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:24.874058 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:24.874969 (+ 911us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:24.875552 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:24.875562 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:24.875569 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:24.876011 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:24.876846 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:24.876859 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:24.877528 (+ 669us) client_negotiation.cc:769] Sending connection context
0923 17:21:24.877897 (+ 369us) client_negotiation.cc:240] Negotiation successful
0923 17:21:24.878625 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":455,"threads_started":1}
I20260923 17:21:24.882093 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:24.850032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:47187 (local address 127.28.72.254:45335)
0923 17:21:24.850593 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:24.850600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:24.852087 (+ 1487us) server_negotiation.cc:407] Connection header received
0923 17:21:24.855167 (+ 3080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:24.855174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:24.855276 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:24.855424 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:24.857958 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:24.859492 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:24.862220 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:24.862930 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:24.863132 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:24.869647 (+ 6515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:24.869658 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:24.869667 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:24.869726 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:24.873798 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:24.875220 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:24.875229 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:24.875234 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:24.875391 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:24.876184 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:24.876193 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:24.876198 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:24.876614 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:24.876785 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:24.881566 (+ 4781us) server_negotiation.cc:299] Negotiation successful
0923 17:21:24.881835 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":306,"threads_started":1}
I20260923 17:21:24.901341 30162 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:45615
I20260923 17:21:24.901388 30297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:45615 every 8 connection(s)
I20260923 17:21:24.902930 30162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:21:24.905218 30298 heartbeater.cc:347] Connected to a master server at 127.28.72.254:45335
I20260923 17:21:24.905488 30298 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:24.906221 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 30162
I20260923 17:21:24.906433 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:21:24.906509 30298 heartbeater.cc:511] Master 127.28.72.254:45335 requested a full tablet report, sending...
I20260923 17:21:24.906527 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:24.888983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:59975 (local address 127.28.72.254:45335)
0923 17:21:24.889248 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:24.889259 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:24.890096 (+ 837us) server_negotiation.cc:407] Connection header received
0923 17:21:24.890466 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:24.890470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:24.890546 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:24.890816 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:24.893038 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:24.894055 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:24.896025 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:24.896952 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:24.897129 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:24.899868 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:24.899873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:24.899877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:24.899913 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:24.903010 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:24.904057 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:24.904066 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:24.904073 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:24.904262 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:24.904840 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:24.904844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:24.904847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:24.905160 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:24.905341 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:24.905804 (+ 463us) server_negotiation.cc:299] Negotiation successful
0923 17:21:24.906116 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 17:21:24.906630 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:24.888948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:45335 (local address 127.28.72.193:59975)
0923 17:21:24.889918 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:24.889949 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:24.890188 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:24.890897 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:24.890905 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:24.892086 (+ 1181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:24.892701 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:24.892726 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:24.894223 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:24.894230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:24.895791 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:24.895838 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:24.897412 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:24.897427 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:24.897657 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:24.898576 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:24.898597 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:24.899635 (+ 1038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:24.903217 (+ 3582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:24.903223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:24.903228 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:24.903849 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:24.904445 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:24.904453 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:24.904458 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:24.904647 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:24.905294 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:24.905299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:24.905525 (+ 226us) client_negotiation.cc:769] Sending connection context
0923 17:21:24.905684 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 17:21:24.906088 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":454,"threads_started":1}
I20260923 17:21:24.907292 28963 external_mini_cluster.cc:1428] Setting key 45f6e1b796fa36795d587253ae5afb04
I20260923 17:21:24.909135 30090 ts_manager.cc:194] Registered new tserver with Master: d25b5fb40d2d4c8b8bb4145e0ee482f8 (127.28.72.193:45615)
I20260923 17:21:24.911087 30090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:47187
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:21:24.951627 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:45335
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:25.292021 30302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:25.292328 30302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:25.292435 30302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:25.299964 30302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:25.300091 30302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:25.300177 30302 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:21:25.300261 30302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:21:25.310647 30302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.72.254:45335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:25.311992 30302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:25.313315 30302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:25.327497 30307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:25.327548 30310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:25.327522 30308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:25.328852 30302 server_base.cc:1034] running on GCE node
I20260923 17:21:25.329512 30302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:25.330683 30302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:25.331981 30302 hybrid_clock.cc:648] HybridClock initialized: now 1790184085331885 us; error 68 us; skew 500 ppm
I20260923 17:21:25.332362 30302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:25 dist-test-slave-j0m4 krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184085, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:25.340372 30302 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:21:25.342341 30302 webserver.cc:460] Webserver started at http://127.28.72.194:40249/ using document root <none> and password file <none>
I20260923 17:21:25.343312 30302 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:25.343484 30302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:25.343861 30302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:25.347262 30302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "1c77fad24351469db34e71b369c423ac"
format_stamp: "Formatted at 2026-09-23 17:21:25 on dist-test-slave-j0m4"
server_key: "d111812bef44c85e14385701d386b10e"
server_key_iv: "16afebdce9ee74ae7949d987423944bd"
server_key_version: "encryptionkey@0"
I20260923 17:21:25.348210 30302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "1c77fad24351469db34e71b369c423ac"
format_stamp: "Formatted at 2026-09-23 17:21:25 on dist-test-slave-j0m4"
server_key: "d111812bef44c85e14385701d386b10e"
server_key_iv: "16afebdce9ee74ae7949d987423944bd"
server_key_version: "encryptionkey@0"
I20260923 17:21:25.354173 30302 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 17:21:25.358379 30317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:25.360073 30302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:21:25.360327 30302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "1c77fad24351469db34e71b369c423ac"
format_stamp: "Formatted at 2026-09-23 17:21:25 on dist-test-slave-j0m4"
server_key: "d111812bef44c85e14385701d386b10e"
server_key_iv: "16afebdce9ee74ae7949d987423944bd"
server_key_version: "encryptionkey@0"
I20260923 17:21:25.360666 30302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:25.378401 30302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:25.383838 30302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:25.384162 30302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:25.387648 30302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:25.387795 30302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:25.388000 30302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:25.388121 30302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:21:25 dist-test-slave-j0m4 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184085, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:25.414585 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:25.387901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:38769 (local address 127.28.72.254:45335)
0923 17:21:25.388094 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:25.388101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:25.389709 (+ 1608us) server_negotiation.cc:407] Connection header received
0923 17:21:25.391967 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:25.391973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:25.392065 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:25.392187 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:25.394332 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.395399 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.398139 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.399039 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.399280 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:25.405398 (+ 6118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:25.405408 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:25.405417 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:25.405472 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:25.409428 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:25.410768 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:25.410778 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:25.410784 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:25.410992 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:25.411936 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:25.411947 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:25.411953 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:25.412458 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:25.412645 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:25.413621 (+ 976us) server_negotiation.cc:299] Negotiation successful
0923 17:21:25.413983 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:21:25.415310 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:25.388598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:45335 (local address 127.28.72.194:38769)
0923 17:21:25.389508 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:25.389560 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:25.391647 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:25.392436 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:25.392445 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:25.393096 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:25.394124 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:25.394146 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.395543 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.395552 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:25.397923 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:25.397946 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.399489 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.399500 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:25.399813 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:25.400787 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:25.400857 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:25.405059 (+ 4202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:25.409635 (+ 4576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:25.409646 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:25.409676 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:25.410582 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:25.411225 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:25.411234 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:25.411242 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:25.411700 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:25.412669 (+ 969us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:25.412684 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:25.413128 (+ 444us) client_negotiation.cc:769] Sending connection context
0923 17:21:25.413399 (+ 271us) client_negotiation.cc:240] Negotiation successful
0923 17:21:25.413846 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":381,"threads_started":1}
I20260923 17:21:25.439229 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:25.422046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:46047 (local address 127.28.72.254:45335)
0923 17:21:25.422269 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:25.422274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:25.423305 (+ 1031us) server_negotiation.cc:407] Connection header received
0923 17:21:25.423467 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:25.423475 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:25.423596 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:25.423774 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:25.425133 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.426136 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.427762 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.428392 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.428633 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:25.431919 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:25.431928 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:25.431937 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:25.432003 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:25.435782 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:25.436908 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:25.436912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:25.436915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:25.437056 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:25.437622 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:25.437629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:25.437633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:25.438028 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:25.438189 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:25.438596 (+ 407us) server_negotiation.cc:299] Negotiation successful
0923 17:21:25.438817 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:21:25.439344 30390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:25.422163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:45335 (local address 127.28.72.194:46047)
0923 17:21:25.423178 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:25.423203 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:25.423330 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:25.423847 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:25.423851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:25.424456 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:25.424959 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:25.424971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.426289 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.426296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:25.427594 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:25.427607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.428701 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.428710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:25.429023 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:25.429843 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:25.429887 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:25.431562 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:25.435936 (+ 4374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:25.435942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:25.435946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:25.436700 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:25.437190 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:25.437195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:25.437199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:25.437351 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:25.438153 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:25.438158 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:25.438345 (+ 187us) client_negotiation.cc:769] Sending connection context
0923 17:21:25.438490 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 17:21:25.438842 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":630,"threads_started":1}
I20260923 17:21:25.450812 30302 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:38479
I20260923 17:21:25.450834 30436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:38479 every 8 connection(s)
I20260923 17:21:25.452303 30302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:21:25.454854 30437 heartbeater.cc:347] Connected to a master server at 127.28.72.254:45335
I20260923 17:21:25.455106 30437 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:25.455864 30437 heartbeater.cc:511] Master 127.28.72.254:45335 requested a full tablet report, sending...
I20260923 17:21:25.457352 30090 ts_manager.cc:194] Registered new tserver with Master: 1c77fad24351469db34e71b369c423ac (127.28.72.194:38479)
I20260923 17:21:25.458181 30090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:38769
I20260923 17:21:25.462817 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 30302
I20260923 17:21:25.462976 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:21:25.463476 28963 external_mini_cluster.cc:1428] Setting key fb3bab01c56ee2743e127d2bf9ac9b24
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:21:25.507675 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:45335
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:25.849550 30441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:25.849879 30441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:25.850045 30441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:25.857736 30441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:25.857878 30441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:25.858035 30441 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:21:25.858157 30441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:21:25.868573 30441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.72.254:45335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:25.869979 30441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:25.871505 30441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:25.886595 30446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:25.886699 30449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:25.887233 30447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:25.887904 30441 server_base.cc:1034] running on GCE node
I20260923 17:21:25.889076 30441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:25.890621 30441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:25.891927 30441 hybrid_clock.cc:648] HybridClock initialized: now 1790184085891826 us; error 53 us; skew 500 ppm
I20260923 17:21:25.892336 30441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:25 dist-test-slave-j0m4 krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184085, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:25.896008 30441 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:21:25.897609 30441 webserver.cc:460] Webserver started at http://127.28.72.195:34889/ using document root <none> and password file <none>
I20260923 17:21:25.898325 30441 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:25.898414 30441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:25.898700 30441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:25.901724 30441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "29a52b5aff154351afd4ae713fb8f6eb"
format_stamp: "Formatted at 2026-09-23 17:21:25 on dist-test-slave-j0m4"
server_key: "1904c706e374acd38f71496a6ba4b78b"
server_key_iv: "978e0dc5e2771fee7184d099aca2c45b"
server_key_version: "encryptionkey@0"
I20260923 17:21:25.902527 30441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "29a52b5aff154351afd4ae713fb8f6eb"
format_stamp: "Formatted at 2026-09-23 17:21:25 on dist-test-slave-j0m4"
server_key: "1904c706e374acd38f71496a6ba4b78b"
server_key_iv: "978e0dc5e2771fee7184d099aca2c45b"
server_key_version: "encryptionkey@0"
I20260923 17:21:25.908027 30441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260923 17:21:25.911898 30456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:25.913506 30441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:21:25.913694 30441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "29a52b5aff154351afd4ae713fb8f6eb"
format_stamp: "Formatted at 2026-09-23 17:21:25 on dist-test-slave-j0m4"
server_key: "1904c706e374acd38f71496a6ba4b78b"
server_key_iv: "978e0dc5e2771fee7184d099aca2c45b"
server_key_version: "encryptionkey@0"
I20260923 17:21:25.914122 30298 heartbeater.cc:503] Master 127.28.72.254:45335 was elected leader, sending a full tablet report...
I20260923 17:21:25.914000 30441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:25.934021 30441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:25.939256 30441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:25.939780 30441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:25.942555 30441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:25.942649 30441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:25.942824 30441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:25.942910 30441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:21:25 dist-test-slave-j0m4 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184085, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:25.972674 30465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:25.943676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:34565 (local address 127.28.72.254:45335)
0923 17:21:25.944185 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:25.944194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:25.945657 (+ 1463us) server_negotiation.cc:407] Connection header received
0923 17:21:25.948376 (+ 2719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:25.948383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:25.948483 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:25.948624 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:25.951260 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.952282 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.955933 (+ 3651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.956366 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.956486 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:25.963591 (+ 7105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:25.963601 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:25.963608 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:25.963664 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:25.968353 (+ 4689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:25.969773 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:25.969778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:25.969781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:25.969937 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:25.970477 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:25.970483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:25.970487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:25.970797 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:25.971007 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:25.971942 (+ 935us) server_negotiation.cc:299] Negotiation successful
0923 17:21:25.972210 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":240,"threads_started":1}
I20260923 17:21:25.973788 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:25.944467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:45335 (local address 127.28.72.195:34565)
0923 17:21:25.945449 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:25.945501 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:25.947896 (+ 2395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:25.948982 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:25.949000 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:25.949956 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:25.951021 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:25.951049 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.952526 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.952537 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:25.955708 (+ 3171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:25.955735 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.956697 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.956707 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:25.957112 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:25.958182 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:25.958247 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:25.963159 (+ 4912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:25.968636 (+ 5477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:25.968654 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:25.968685 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:25.969532 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:25.970076 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:25.970081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:25.970087 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:25.970329 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:25.970955 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:25.970965 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:25.971506 (+ 541us) client_negotiation.cc:769] Sending connection context
0923 17:21:25.971798 (+ 292us) client_negotiation.cc:240] Negotiation successful
0923 17:21:25.972426 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":435,"threads_started":1}
I20260923 17:21:25.997294 30441 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:45473
I20260923 17:21:25.997313 30576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:45473 every 8 connection(s)
I20260923 17:21:25.998988 30465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:25.982194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:57097 (local address 127.28.72.254:45335)
0923 17:21:25.982386 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:25.982391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:25.982948 (+ 557us) server_negotiation.cc:407] Connection header received
0923 17:21:25.983188 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:25.983196 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:25.983370 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:25.983491 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:25.985110 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.986066 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.988232 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.988686 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.988840 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:25.992276 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:25.992286 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:25.992293 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:25.992349 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:25.996037 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:25.996926 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:25.996934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:25.996938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:25.997080 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:25.997529 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:25.997537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:25.997540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:25.997834 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:25.997961 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:25.998506 (+ 545us) server_negotiation.cc:299] Negotiation successful
0923 17:21:25.998698 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:21:25.999182 30441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:21:25.999228 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:25.982157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:45335 (local address 127.28.72.195:57097)
0923 17:21:25.982823 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:25.982841 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:25.983027 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:25.983526 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:25.983531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:25.984287 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:25.984889 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:25.984906 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.986220 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.986230 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:25.988055 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:25.988070 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:25.988926 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:25.988932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:25.989125 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:25.989990 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:25.990010 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:25.991913 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:25.996208 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:25.996216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:25.996222 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:25.996763 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:25.997204 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:25.997209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:25.997212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:25.997351 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:25.997978 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:25.997983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:25.998200 (+ 217us) client_negotiation.cc:769] Sending connection context
0923 17:21:25.998374 (+ 174us) client_negotiation.cc:240] Negotiation successful
0923 17:21:25.998742 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":375,"threads_started":1}
I20260923 17:21:26.001597 30577 heartbeater.cc:347] Connected to a master server at 127.28.72.254:45335
I20260923 17:21:26.001895 30577 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:26.002885 30577 heartbeater.cc:511] Master 127.28.72.254:45335 requested a full tablet report, sending...
I20260923 17:21:26.004494 30090 ts_manager.cc:194] Registered new tserver with Master: 29a52b5aff154351afd4ae713fb8f6eb (127.28.72.195:45473)
I20260923 17:21:26.005357 30090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:34565
I20260923 17:21:26.008045 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 30441
I20260923 17:21:26.008258 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:21:26.008822 28963 external_mini_cluster.cc:1428] Setting key 332eed2cc95e86f9a55b6340418e9da1
I20260923 17:21:26.012614 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:21:26 dist-test-slave-j0m4 krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184086, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:21:26 dist-test-slave-j0m4 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184086, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:26.064283 30465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.048737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.28.72.254:45335)
0923 17:21:26.048930 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:26.048935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:26.049265 (+ 330us) server_negotiation.cc:407] Connection header received
0923 17:21:26.049460 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:26.049465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:26.049534 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:26.049633 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:26.051191 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.052036 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.053404 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.053835 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.053937 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.058144 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:26.058149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:26.058153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:26.058194 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:26.061148 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:26.061994 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:26.061999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:26.062002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:26.062219 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:26.062730 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:26.062739 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:26.062744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:26.063068 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:26.063391 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:26.063733 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 17:21:26.063954 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 17:21:26.070284 30090 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:21:26.078131 30090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:21:26.121613 30593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.093200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:38479 (local address 127.0.0.1:60180)
0923 17:21:26.104060 (+ 10860us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:26.104086 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:26.104276 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:26.110640 (+ 6364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:26.110650 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:26.110694 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:26.113082 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:26.113104 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.114779 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.114788 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:26.118203 (+ 3415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:26.118221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.120378 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.120384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:26.120607 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.120674 (+ 67us) client_negotiation.cc:769] Sending connection context
0923 17:21:26.120862 (+ 188us) client_negotiation.cc:240] Negotiation successful
0923 17:21:26.121108 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10640,"mutex_wait_us":137,"thread_start_us":17627,"threads_started":1}
I20260923 17:21:26.121685 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.094489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.28.72.194:38479)
0923 17:21:26.109572 (+ 15083us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:26.109585 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:26.109630 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:21:26.109796 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:26.109811 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:26.110221 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:26.110560 (+ 339us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:26.113372 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.114612 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.118420 (+ 3808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.120127 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.120319 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.120599 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:26.121193 (+ 594us) server_negotiation.cc:299] Negotiation successful
0923 17:21:26.121391 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14836,"thread_start_us":14684,"threads_started":1}
I20260923 17:21:26.121691 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.092900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:45473 (local address 127.0.0.1:59278)
0923 17:21:26.094050 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:26.094101 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:26.094309 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:26.112704 (+ 18395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:26.112713 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:26.112756 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:26.113590 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:26.113614 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.116286 (+ 2672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.116291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:26.118881 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:26.118910 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.120848 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.120856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:26.121021 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.121054 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:21:26.121143 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 17:21:26.121298 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":4,"thread_start_us":730,"threads_started":1}
I20260923 17:21:26.122385 30592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.093233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.28.72.195:45473)
0923 17:21:26.111818 (+ 18585us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:26.111833 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:26.111880 (+ 47us) server_negotiation.cc:407] Connection header received
0923 17:21:26.112019 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:26.112032 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:26.112388 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:26.113915 (+ 1527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:26.114307 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.116022 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.119173 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.120554 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.120767 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.121031 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:26.121848 (+ 817us) server_negotiation.cc:299] Negotiation successful
0923 17:21:26.122032 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18126,"thread_start_us":18020,"threads_started":1}
I20260923 17:21:26.123533 30595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.094876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.28.72.193:45615)
0923 17:21:26.114289 (+ 19413us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:26.114298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:26.114330 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:21:26.114445 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:26.114455 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:26.114757 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:26.115812 (+ 1055us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:26.116035 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.117426 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.120195 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.121864 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.122620 (+ 756us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.122843 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:26.122954 (+ 111us) server_negotiation.cc:299] Negotiation successful
0923 17:21:26.123225 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19199,"thread_start_us":19915,"threads_started":1}
I20260923 17:21:26.123103 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.093979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:45615 (local address 127.0.0.1:59528)
0923 17:21:26.108576 (+ 14597us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:26.108610 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:26.108905 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:26.115038 (+ 6133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:26.115043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:26.115077 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:26.115579 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:26.115595 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.117603 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.117608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:26.119998 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:26.120017 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.122163 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.122171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:26.122411 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.122447 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:21:26.122520 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 17:21:26.122667 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14330,"mutex_wait_us":4,"thread_start_us":9678,"threads_started":1}
I20260923 17:21:26.126986 30507 tablet_service.cc:1467] Processing CreateTablet for tablet a8cbe68462d94e688c6d7979132331f4 (DEFAULT_TABLE table=test-table [id=783d2bddb6c64931ad94a13c8104d43c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:26.127326 30367 tablet_service.cc:1467] Processing CreateTablet for tablet a8cbe68462d94e688c6d7979132331f4 (DEFAULT_TABLE table=test-table [id=783d2bddb6c64931ad94a13c8104d43c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:26.128531 30507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8cbe68462d94e688c6d7979132331f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:26.128036 30228 tablet_service.cc:1467] Processing CreateTablet for tablet a8cbe68462d94e688c6d7979132331f4 (DEFAULT_TABLE table=test-table [id=783d2bddb6c64931ad94a13c8104d43c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:26.128921 30367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8cbe68462d94e688c6d7979132331f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:26.129571 30228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8cbe68462d94e688c6d7979132331f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:26.141120 30596 tablet_bootstrap.cc:492] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb: Bootstrap starting.
I20260923 17:21:26.144538 30596 tablet_bootstrap.cc:654] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:26.145920 30596 log.cc:824] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:26.146411 30597 tablet_bootstrap.cc:492] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac: Bootstrap starting.
I20260923 17:21:26.149852 30599 tablet_bootstrap.cc:492] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8: Bootstrap starting.
I20260923 17:21:26.151639 30597 tablet_bootstrap.cc:654] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:26.153343 30597 log.cc:824] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:26.154182 30596 tablet_bootstrap.cc:492] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb: No bootstrap required, opened a new log
I20260923 17:21:26.154234 30599 tablet_bootstrap.cc:654] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:26.154596 30596 ts_tablet_manager.cc:1397] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260923 17:21:26.155655 30599 log.cc:824] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:26.158439 30599 tablet_bootstrap.cc:492] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8: No bootstrap required, opened a new log
I20260923 17:21:26.158486 30597 tablet_bootstrap.cc:492] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac: No bootstrap required, opened a new log
I20260923 17:21:26.158743 30599 ts_tablet_manager.cc:1397] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 17:21:26.158748 30597 ts_tablet_manager.cc:1397] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20260923 17:21:26.166332 30596 raft_consensus.cc:348] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } }
I20260923 17:21:26.167064 30596 raft_consensus.cc:374] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:26.167255 30596 raft_consensus.cc:729] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a52b5aff154351afd4ae713fb8f6eb, State: Initialized, Role: FOLLOWER
I20260923 17:21:26.168082 30596 consensus_queue.cc:260] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } }
I20260923 17:21:26.168048 30599 raft_consensus.cc:348] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } }
I20260923 17:21:26.168443 30599 raft_consensus.cc:374] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:26.168609 30599 raft_consensus.cc:729] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25b5fb40d2d4c8b8bb4145e0ee482f8, State: Initialized, Role: FOLLOWER
I20260923 17:21:26.169312 30577 heartbeater.cc:503] Master 127.28.72.254:45335 was elected leader, sending a full tablet report...
I20260923 17:21:26.169469 30599 consensus_queue.cc:260] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } }
I20260923 17:21:26.170029 30596 ts_tablet_manager.cc:1428] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260923 17:21:26.171437 30599 ts_tablet_manager.cc:1428] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8: Time spent starting tablet: real 0.012s user 0.010s sys 0.004s
I20260923 17:21:26.171305 30597 raft_consensus.cc:348] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } }
I20260923 17:21:26.171872 30597 raft_consensus.cc:374] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:26.171995 30597 raft_consensus.cc:729] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c77fad24351469db34e71b369c423ac, State: Initialized, Role: FOLLOWER
I20260923 17:21:26.172626 30597 consensus_queue.cc:260] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } }
I20260923 17:21:26.174108 30437 heartbeater.cc:503] Master 127.28.72.254:45335 was elected leader, sending a full tablet report...
I20260923 17:21:26.174439 30597 ts_tablet_manager.cc:1428] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260923 17:21:26.181017 30604 raft_consensus.cc:482] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:21:26.181308 30604 raft_consensus.cc:504] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } }
I20260923 17:21:26.183301 30604 leader_election.cc:288] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29a52b5aff154351afd4ae713fb8f6eb (127.28.72.195:45473), d25b5fb40d2d4c8b8bb4145e0ee482f8 (127.28.72.193:45615)
I20260923 17:21:26.191710 30592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.184635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:57095 (local address 127.28.72.195:45473)
0923 17:21:26.184875 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:26.184883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:26.185749 (+ 866us) server_negotiation.cc:407] Connection header received
0923 17:21:26.185948 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:26.185956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:26.186046 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:26.186114 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:26.186784 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.187618 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.189748 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.190597 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.190803 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.190877 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:26.191290 (+ 413us) server_negotiation.cc:299] Negotiation successful
0923 17:21:26.191434 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260923 17:21:26.191709 30595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.183700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:35063 (local address 127.28.72.193:45615)
0923 17:21:26.183930 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:26.183938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:26.185835 (+ 1897us) server_negotiation.cc:407] Connection header received
0923 17:21:26.186112 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:26.186118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:26.186193 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:26.186298 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:26.186966 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.187815 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.189820 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.190583 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.190756 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.190850 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:26.191295 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 17:21:26.191425 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:21:26.191907 30605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.184123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:45615 (local address 127.28.72.194:35063)
0923 17:21:26.185742 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:26.185761 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:26.185905 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:26.186328 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:26.186331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:26.186418 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:26.186792 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:26.186804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.188002 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.188006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:26.189626 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:26.189638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.190838 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.190844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:26.191008 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.191101 (+ 93us) client_negotiation.cc:769] Sending connection context
0923 17:21:26.191259 (+ 158us) client_negotiation.cc:240] Negotiation successful
0923 17:21:26.191434 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1458,"mutex_wait_us":84,"spinlock_wait_cycles":26880,"thread_start_us":920,"threads_started":1}
I20260923 17:21:26.192368 30606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.183737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:45473 (local address 127.28.72.194:57095)
0923 17:21:26.185625 (+ 1888us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:26.185648 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:26.185806 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:26.186150 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:26.186155 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:26.186179 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:26.186622 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:26.186634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.187748 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.187753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:26.189579 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:26.189596 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.190857 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.190865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:26.191046 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.191098 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 17:21:26.191263 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 17:21:26.191425 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1732,"mutex_wait_us":140,"thread_start_us":1596,"threads_started":1}
I20260923 17:21:26.193068 30248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cbe68462d94e688c6d7979132331f4" candidate_uuid: "1c77fad24351469db34e71b369c423ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" is_pre_election: true
I20260923 17:21:26.193377 30527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cbe68462d94e688c6d7979132331f4" candidate_uuid: "1c77fad24351469db34e71b369c423ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a52b5aff154351afd4ae713fb8f6eb" is_pre_election: true
I20260923 17:21:26.193902 30527 raft_consensus.cc:2454] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c77fad24351469db34e71b369c423ac in term 0.
I20260923 17:21:26.194041 30248 raft_consensus.cc:2454] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c77fad24351469db34e71b369c423ac in term 0.
I20260923 17:21:26.194729 30320 leader_election.cc:302] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c77fad24351469db34e71b369c423ac, 29a52b5aff154351afd4ae713fb8f6eb, d25b5fb40d2d4c8b8bb4145e0ee482f8; no voters:
I20260923 17:21:26.195250 30604 raft_consensus.cc:2781] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:21:26.195418 30604 raft_consensus.cc:482] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:21:26.195531 30604 raft_consensus.cc:3037] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:26.197772 30604 raft_consensus.cc:504] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } }
I20260923 17:21:26.198459 30604 leader_election.cc:288] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [CANDIDATE]: Term 1 election: Requested vote from peers 29a52b5aff154351afd4ae713fb8f6eb (127.28.72.195:45473), d25b5fb40d2d4c8b8bb4145e0ee482f8 (127.28.72.193:45615)
I20260923 17:21:26.199048 30527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cbe68462d94e688c6d7979132331f4" candidate_uuid: "1c77fad24351469db34e71b369c423ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a52b5aff154351afd4ae713fb8f6eb"
I20260923 17:21:26.199066 30248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cbe68462d94e688c6d7979132331f4" candidate_uuid: "1c77fad24351469db34e71b369c423ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8"
I20260923 17:21:26.199347 30248 raft_consensus.cc:3037] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:26.199343 30527 raft_consensus.cc:3037] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:26.201620 30248 raft_consensus.cc:2454] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c77fad24351469db34e71b369c423ac in term 1.
I20260923 17:21:26.201690 30527 raft_consensus.cc:2454] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c77fad24351469db34e71b369c423ac in term 1.
I20260923 17:21:26.202176 30321 leader_election.cc:302] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c77fad24351469db34e71b369c423ac, d25b5fb40d2d4c8b8bb4145e0ee482f8; no voters:
I20260923 17:21:26.202575 30604 raft_consensus.cc:2781] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:26.203081 30604 raft_consensus.cc:686] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [term 1 LEADER]: Becoming Leader. State: Replica: 1c77fad24351469db34e71b369c423ac, State: Running, Role: LEADER
I20260923 17:21:26.203732 30604 consensus_queue.cc:237] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } }
W20260923 17:21:26.204442 30438 tablet.cc:2369] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:21:26.208727 30089 catalog_manager.cc:5495] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c77fad24351469db34e71b369c423ac (127.28.72.194). New cstate: current_term: 1 leader_uuid: "1c77fad24351469db34e71b369c423ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } health_report { overall_health: UNKNOWN } } }
I20260923 17:21:26.250097 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.243800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.28.72.194:38479)
0923 17:21:26.244074 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:26.244080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:26.244205 (+ 125us) server_negotiation.cc:407] Connection header received
0923 17:21:26.244661 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:26.244665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:26.244744 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:26.244944 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:26.245316 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.246194 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.247696 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.248119 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.248259 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.248841 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:26.249423 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:26.249583 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:26.249725 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 17:21:26.249812 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260923 17:21:26.251097 30578 tablet.cc:2369] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:21:26.258787 30248 raft_consensus.cc:1261] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 1 FOLLOWER]: Refusing update from remote peer 1c77fad24351469db34e71b369c423ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:26.259233 30527 raft_consensus.cc:1261] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb [term 1 FOLLOWER]: Refusing update from remote peer 1c77fad24351469db34e71b369c423ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:26.259886 30607 consensus_queue.cc:1035] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:26.260268 30604 consensus_queue.cc:1035] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:26.272768 30610 mvcc.cc:205] Tried to move back new op lower bound from 7332594017305886720 to 7332594017095753728. Current Snapshot: MvccSnapshot[applied={T|T < 7332594017305886720}]
I20260923 17:21:26.301190 30089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:26.301493 30089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:26.305236 30089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38: Sending DeleteTablet for 3 replicas of tablet a8cbe68462d94e688c6d7979132331f4
I20260923 17:21:26.306476 30507 tablet_service.cc:1514] Processing DeleteTablet for tablet a8cbe68462d94e688c6d7979132331f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:26 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:59278
I20260923 17:21:26.306607 30228 tablet_service.cc:1514] Processing DeleteTablet for tablet a8cbe68462d94e688c6d7979132331f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:26 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:59528
I20260923 17:21:26.306605 30367 tablet_service.cc:1514] Processing DeleteTablet for tablet a8cbe68462d94e688c6d7979132331f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:26 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:60180
I20260923 17:21:26.311286 30089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:21:26.313446 30089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:21:26.322383 30620 tablet_replica.cc:330] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb: stopping tablet replica
I20260923 17:21:26.323199 30620 raft_consensus.cc:2229] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:26.323942 30620 raft_consensus.cc:2258] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:26.325577 30507 tablet_service.cc:1467] Processing CreateTablet for tablet 830f22814f004b6984d596b72320849c (DEFAULT_TABLE table=test-table [id=ee0023745b31438e8b809f246c236c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:26.325541 30227 tablet_service.cc:1467] Processing CreateTablet for tablet 830f22814f004b6984d596b72320849c (DEFAULT_TABLE table=test-table [id=ee0023745b31438e8b809f246c236c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:26.326129 30507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830f22814f004b6984d596b72320849c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:26.326144 30227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830f22814f004b6984d596b72320849c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:26.326161 30366 tablet_service.cc:1467] Processing CreateTablet for tablet 830f22814f004b6984d596b72320849c (DEFAULT_TABLE table=test-table [id=ee0023745b31438e8b809f246c236c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:26.326778 30366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830f22814f004b6984d596b72320849c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:26.333657 30599 tablet_bootstrap.cc:492] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8: Bootstrap starting.
I20260923 17:21:26.334473 30620 ts_tablet_manager.cc:1905] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:26.336074 30599 tablet_bootstrap.cc:654] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:26.336494 30596 tablet_bootstrap.cc:492] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb: Bootstrap starting.
I20260923 17:21:26.338267 30596 tablet_bootstrap.cc:654] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:26.339756 30597 tablet_bootstrap.cc:492] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac: Bootstrap starting.
I20260923 17:21:26.341977 30597 tablet_bootstrap.cc:654] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:26.344923 30596 tablet_bootstrap.cc:492] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb: No bootstrap required, opened a new log
I20260923 17:21:26.345130 30596 ts_tablet_manager.cc:1397] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20260923 17:21:26.346172 30596 raft_consensus.cc:348] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.346478 30596 raft_consensus.cc:374] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:26.346568 30596 raft_consensus.cc:729] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a52b5aff154351afd4ae713fb8f6eb, State: Initialized, Role: FOLLOWER
I20260923 17:21:26.346987 30596 consensus_queue.cc:260] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.348608 30621 tablet_replica.cc:330] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8: stopping tablet replica
I20260923 17:21:26.348601 30596 ts_tablet_manager.cc:1428] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260923 17:21:26.349208 30622 tablet_replica.cc:330] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac: stopping tablet replica
I20260923 17:21:26.349742 30622 raft_consensus.cc:2229] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:21:26.350564 30622 raft_consensus.cc:2258] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:26.352010 30621 raft_consensus.cc:2229] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:26.352526 30621 raft_consensus.cc:2258] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:26.350831 30599 tablet_bootstrap.cc:492] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8: No bootstrap required, opened a new log
I20260923 17:21:26.353021 30599 ts_tablet_manager.cc:1397] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.008s
I20260923 17:21:26.353318 30622 ts_tablet_manager.cc:1905] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:26.354382 30599 raft_consensus.cc:348] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.354646 30599 raft_consensus.cc:374] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:26.354736 30599 raft_consensus.cc:729] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25b5fb40d2d4c8b8bb4145e0ee482f8, State: Initialized, Role: FOLLOWER
I20260923 17:21:26.355006 30599 consensus_queue.cc:260] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.355616 30599 ts_tablet_manager.cc:1428] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:21:26.357612 30621 ts_tablet_manager.cc:1905] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:26.363919 30622 ts_tablet_manager.cc:1918] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:21:26.364130 30622 log.cc:1196] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/a8cbe68462d94e688c6d7979132331f4
I20260923 17:21:26.366425 30620 ts_tablet_manager.cc:1918] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:21:26.366633 30620 log.cc:1196] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/a8cbe68462d94e688c6d7979132331f4
I20260923 17:21:26.367208 30620 ts_tablet_manager.cc:1939] T a8cbe68462d94e688c6d7979132331f4 P 29a52b5aff154351afd4ae713fb8f6eb: Deleting consensus metadata
I20260923 17:21:26.367198 30622 ts_tablet_manager.cc:1939] T a8cbe68462d94e688c6d7979132331f4 P 1c77fad24351469db34e71b369c423ac: Deleting consensus metadata
I20260923 17:21:26.369189 30077 catalog_manager.cc:4841] TS 29a52b5aff154351afd4ae713fb8f6eb (127.28.72.195:45473): tablet a8cbe68462d94e688c6d7979132331f4 (table test-table [id=783d2bddb6c64931ad94a13c8104d43c]) successfully deleted
I20260923 17:21:26.370281 30075 catalog_manager.cc:4841] TS 1c77fad24351469db34e71b369c423ac (127.28.72.194:38479): tablet a8cbe68462d94e688c6d7979132331f4 (table test-table [id=783d2bddb6c64931ad94a13c8104d43c]) successfully deleted
I20260923 17:21:26.370923 30621 ts_tablet_manager.cc:1918] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:21:26.371008 30597 tablet_bootstrap.cc:492] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac: No bootstrap required, opened a new log
I20260923 17:21:26.371160 30621 log.cc:1196] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/a8cbe68462d94e688c6d7979132331f4
I20260923 17:21:26.371255 30597 ts_tablet_manager.cc:1397] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.006s
I20260923 17:21:26.371711 30621 ts_tablet_manager.cc:1939] T a8cbe68462d94e688c6d7979132331f4 P d25b5fb40d2d4c8b8bb4145e0ee482f8: Deleting consensus metadata
I20260923 17:21:26.372123 30597 raft_consensus.cc:348] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.372386 30597 raft_consensus.cc:374] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:26.372752 30597 raft_consensus.cc:729] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c77fad24351469db34e71b369c423ac, State: Initialized, Role: FOLLOWER
I20260923 17:21:26.373096 30597 consensus_queue.cc:260] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.373472 30078 catalog_manager.cc:4841] TS d25b5fb40d2d4c8b8bb4145e0ee482f8 (127.28.72.193:45615): tablet a8cbe68462d94e688c6d7979132331f4 (table test-table [id=783d2bddb6c64931ad94a13c8104d43c]) successfully deleted
I20260923 17:21:26.374192 30597 ts_tablet_manager.cc:1428] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260923 17:21:26.406178 30299 tablet.cc:2369] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:21:26.468645 30604 raft_consensus.cc:482] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:21:26.468874 30604 raft_consensus.cc:504] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.469718 30604 leader_election.cc:288] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d25b5fb40d2d4c8b8bb4145e0ee482f8 (127.28.72.193:45615), 29a52b5aff154351afd4ae713fb8f6eb (127.28.72.195:45473)
I20260923 17:21:26.470299 30248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830f22814f004b6984d596b72320849c" candidate_uuid: "1c77fad24351469db34e71b369c423ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" is_pre_election: true
I20260923 17:21:26.470453 30527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830f22814f004b6984d596b72320849c" candidate_uuid: "1c77fad24351469db34e71b369c423ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a52b5aff154351afd4ae713fb8f6eb" is_pre_election: true
I20260923 17:21:26.470549 30248 raft_consensus.cc:2454] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c77fad24351469db34e71b369c423ac in term 0.
I20260923 17:21:26.470690 30527 raft_consensus.cc:2454] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c77fad24351469db34e71b369c423ac in term 0.
I20260923 17:21:26.471274 30320 leader_election.cc:302] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c77fad24351469db34e71b369c423ac, 29a52b5aff154351afd4ae713fb8f6eb, d25b5fb40d2d4c8b8bb4145e0ee482f8; no voters:
I20260923 17:21:26.471601 30604 raft_consensus.cc:2781] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:21:26.471728 30604 raft_consensus.cc:482] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:21:26.471832 30604 raft_consensus.cc:3037] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:26.475760 30604 raft_consensus.cc:504] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.477205 30248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830f22814f004b6984d596b72320849c" candidate_uuid: "1c77fad24351469db34e71b369c423ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8"
I20260923 17:21:26.477449 30248 raft_consensus.cc:3037] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:26.478188 30527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830f22814f004b6984d596b72320849c" candidate_uuid: "1c77fad24351469db34e71b369c423ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a52b5aff154351afd4ae713fb8f6eb"
I20260923 17:21:26.478438 30527 raft_consensus.cc:3037] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:26.481380 30248 raft_consensus.cc:2454] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c77fad24351469db34e71b369c423ac in term 1.
I20260923 17:21:26.481547 30527 raft_consensus.cc:2454] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c77fad24351469db34e71b369c423ac in term 1.
I20260923 17:21:26.482007 30321 leader_election.cc:302] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c77fad24351469db34e71b369c423ac, d25b5fb40d2d4c8b8bb4145e0ee482f8; no voters:
I20260923 17:21:26.482232 30604 leader_election.cc:288] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [CANDIDATE]: Term 1 election: Requested vote from peers d25b5fb40d2d4c8b8bb4145e0ee482f8 (127.28.72.193:45615), 29a52b5aff154351afd4ae713fb8f6eb (127.28.72.195:45473)
I20260923 17:21:26.482336 30607 raft_consensus.cc:2781] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:26.482522 30607 raft_consensus.cc:686] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [term 1 LEADER]: Becoming Leader. State: Replica: 1c77fad24351469db34e71b369c423ac, State: Running, Role: LEADER
I20260923 17:21:26.482776 30607 consensus_queue.cc:237] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.485976 30089 catalog_manager.cc:5495] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c77fad24351469db34e71b369c423ac (127.28.72.194). New cstate: current_term: 1 leader_uuid: "1c77fad24351469db34e71b369c423ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } health_report { overall_health: UNKNOWN } } }
Sep 23 17:21:26 dist-test-slave-j0m4 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184084, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:26.554462 30465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.537795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34526 (local address 127.28.72.254:45335)
0923 17:21:26.538024 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:26.538032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:26.538735 (+ 703us) server_negotiation.cc:407] Connection header received
0923 17:21:26.538893 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:26.538899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:26.538989 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:26.539127 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:26.540694 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.542321 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.543761 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.544432 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.544573 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.548474 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:26.548479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:26.548482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:26.548513 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:26.551211 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:26.552184 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:26.552188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:26.552190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:26.552396 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:26.552865 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:26.552872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:26.552880 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:26.553167 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:26.553280 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:26.553811 (+ 531us) server_negotiation.cc:299] Negotiation successful
0923 17:21:26.554178 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":178,"server-negotiator.queue_time_us":85}
I20260923 17:21:26.554615 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.537822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:45335 (local address 127.0.0.1:34526)
0923 17:21:26.538567 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:26.538583 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:26.538768 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:26.539215 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:26.539219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:26.540253 (+ 1034us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:26.540538 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:26.540547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.542472 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.542476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:26.543587 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:26.543599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.544636 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.544641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:26.544767 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.545607 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:26.545673 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:26.548223 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:26.551361 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:26.551376 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:26.551384 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:26.551976 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:26.552523 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:26.552527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:26.552530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:26.552689 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:26.553311 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:26.553320 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:26.553520 (+ 200us) client_negotiation.cc:769] Sending connection context
0923 17:21:26.553695 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 17:21:26.554044 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":420,"threads_started":1}
I20260923 17:21:26.560707 30089 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:34526:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:21:26.561960 30089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:21:26.570355 30228 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba92752854e47ccb0cf61ac9448a42a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7eefe926a5b749efad7b9fe7ab9e6345]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:21:26.570739 30507 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba92752854e47ccb0cf61ac9448a42a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7eefe926a5b749efad7b9fe7ab9e6345]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:21:26.571007 30228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba92752854e47ccb0cf61ac9448a42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:26.571043 30367 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba92752854e47ccb0cf61ac9448a42a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7eefe926a5b749efad7b9fe7ab9e6345]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:21:26.571313 30507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba92752854e47ccb0cf61ac9448a42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:26.571672 30367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba92752854e47ccb0cf61ac9448a42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:26.579700 30596 tablet_bootstrap.cc:492] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb: Bootstrap starting.
I20260923 17:21:26.579700 30599 tablet_bootstrap.cc:492] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8: Bootstrap starting.
I20260923 17:21:26.580992 30597 tablet_bootstrap.cc:492] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac: Bootstrap starting.
I20260923 17:21:26.581640 30596 tablet_bootstrap.cc:654] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:26.582967 30597 tablet_bootstrap.cc:654] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:26.583891 30599 tablet_bootstrap.cc:654] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:26.585466 30596 tablet_bootstrap.cc:492] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb: No bootstrap required, opened a new log
I20260923 17:21:26.585638 30596 ts_tablet_manager.cc:1397] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260923 17:21:26.586035 30597 tablet_bootstrap.cc:492] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac: No bootstrap required, opened a new log
I20260923 17:21:26.586251 30597 ts_tablet_manager.cc:1397] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:21:26.586530 30596 raft_consensus.cc:348] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.586858 30596 raft_consensus.cc:374] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:26.586962 30596 raft_consensus.cc:729] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a52b5aff154351afd4ae713fb8f6eb, State: Initialized, Role: FOLLOWER
I20260923 17:21:26.587303 30596 consensus_queue.cc:260] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.587529 30597 raft_consensus.cc:348] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.587795 30597 raft_consensus.cc:374] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:26.587875 30597 raft_consensus.cc:729] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c77fad24351469db34e71b369c423ac, State: Initialized, Role: FOLLOWER
I20260923 17:21:26.588110 30596 ts_tablet_manager.cc:1428] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:21:26.588176 30597 consensus_queue.cc:260] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.588649 30607 tablet_replica.cc:439] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } }
I20260923 17:21:26.588953 30607 tablet_replica.cc:442] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:21:26.589680 30602 tablet_replica.cc:439] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } }
I20260923 17:21:26.590097 30602 tablet_replica.cc:442] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:21:26.592610 30597 ts_tablet_manager.cc:1428] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260923 17:21:26.596547 30599 tablet_bootstrap.cc:492] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8: No bootstrap required, opened a new log
I20260923 17:21:26.596755 30599 ts_tablet_manager.cc:1397] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.003s
I20260923 17:21:26.597776 30599 raft_consensus.cc:348] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.598009 30599 raft_consensus.cc:374] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:26.598093 30599 raft_consensus.cc:729] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25b5fb40d2d4c8b8bb4145e0ee482f8, State: Initialized, Role: FOLLOWER
I20260923 17:21:26.598395 30599 consensus_queue.cc:260] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.598862 30603 tablet_replica.cc:439] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } }
I20260923 17:21:26.599323 30603 tablet_replica.cc:442] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:21:26.599797 30599 ts_tablet_manager.cc:1428] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260923 17:21:26.868189 30607 raft_consensus.cc:482] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:21:26.868427 30607 raft_consensus.cc:504] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.869129 30607 leader_election.cc:288] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d25b5fb40d2d4c8b8bb4145e0ee482f8 (127.28.72.193:45615), 29a52b5aff154351afd4ae713fb8f6eb (127.28.72.195:45473)
I20260923 17:21:26.869761 30248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba92752854e47ccb0cf61ac9448a42a" candidate_uuid: "1c77fad24351469db34e71b369c423ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" is_pre_election: true
I20260923 17:21:26.869769 30527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba92752854e47ccb0cf61ac9448a42a" candidate_uuid: "1c77fad24351469db34e71b369c423ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a52b5aff154351afd4ae713fb8f6eb" is_pre_election: true
I20260923 17:21:26.870045 30248 raft_consensus.cc:2454] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c77fad24351469db34e71b369c423ac in term 0.
I20260923 17:21:26.870045 30527 raft_consensus.cc:2454] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c77fad24351469db34e71b369c423ac in term 0.
I20260923 17:21:26.870573 30321 leader_election.cc:302] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c77fad24351469db34e71b369c423ac, d25b5fb40d2d4c8b8bb4145e0ee482f8; no voters:
I20260923 17:21:26.870921 30607 raft_consensus.cc:2781] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:21:26.871007 30607 raft_consensus.cc:482] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:21:26.871167 30607 raft_consensus.cc:3037] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:26.873343 30607 raft_consensus.cc:504] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.873981 30607 leader_election.cc:288] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [CANDIDATE]: Term 1 election: Requested vote from peers d25b5fb40d2d4c8b8bb4145e0ee482f8 (127.28.72.193:45615), 29a52b5aff154351afd4ae713fb8f6eb (127.28.72.195:45473)
I20260923 17:21:26.874698 30248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba92752854e47ccb0cf61ac9448a42a" candidate_uuid: "1c77fad24351469db34e71b369c423ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8"
I20260923 17:21:26.874691 30527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba92752854e47ccb0cf61ac9448a42a" candidate_uuid: "1c77fad24351469db34e71b369c423ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a52b5aff154351afd4ae713fb8f6eb"
I20260923 17:21:26.874941 30248 raft_consensus.cc:3037] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:26.875097 30527 raft_consensus.cc:3037] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:26.877568 30527 raft_consensus.cc:2454] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c77fad24351469db34e71b369c423ac in term 1.
I20260923 17:21:26.877993 30248 raft_consensus.cc:2454] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c77fad24351469db34e71b369c423ac in term 1.
I20260923 17:21:26.878134 30320 leader_election.cc:302] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c77fad24351469db34e71b369c423ac, 29a52b5aff154351afd4ae713fb8f6eb; no voters:
I20260923 17:21:26.878512 30607 raft_consensus.cc:2781] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:26.878693 30607 raft_consensus.cc:686] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [term 1 LEADER]: Becoming Leader. State: Replica: 1c77fad24351469db34e71b369c423ac, State: Running, Role: LEADER
I20260923 17:21:26.878995 30607 consensus_queue.cc:237] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } }
I20260923 17:21:26.880829 30604 tablet_replica.cc:439] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac: TxnStatusTablet state changed. Reason: New leader 1c77fad24351469db34e71b369c423ac. Latest consensus state: current_term: 1 leader_uuid: "1c77fad24351469db34e71b369c423ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } }
I20260923 17:21:26.881096 30604 tablet_replica.cc:442] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:21:26.881676 30637 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:21:26.881924 30637 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:21:26.882231 30091 catalog_manager.cc:5495] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c77fad24351469db34e71b369c423ac (127.28.72.194). New cstate: current_term: 1 leader_uuid: "1c77fad24351469db34e71b369c423ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } health_report { overall_health: UNKNOWN } } }
I20260923 17:21:26.927668 30604 consensus_queue.cc:1035] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:26.936471 30604 consensus_queue.cc:1035] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:26.962779 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.956835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:38479 (local address 127.0.0.1:60208)
0923 17:21:26.957101 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:26.957125 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:26.957704 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:26.958267 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:26.958272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:26.958286 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:21:26.958474 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:26.958483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.959679 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.959685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:26.960743 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:26.960755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.961531 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.961536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:26.961695 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.961758 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:21:26.962169 (+ 411us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:21:26.962195 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:21:26.962298 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 17:21:26.962462 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 17:21:26.962852 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:26.956891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.28.72.194:38479)
0923 17:21:26.957976 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:26.957980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:26.957998 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:21:26.958055 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:26.958059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:26.958123 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:26.958232 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:26.958642 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.959543 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.960899 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:26.961356 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:26.961512 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:26.961960 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:26.962067 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:26.962166 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:26.962340 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 17:21:26.962429 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":747,"threads_started":1}
I20260923 17:21:26.967417 30248 raft_consensus.cc:1261] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 1 FOLLOWER]: Refusing update from remote peer 1c77fad24351469db34e71b369c423ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:26.967411 30527 raft_consensus.cc:1261] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb [term 1 FOLLOWER]: Refusing update from remote peer 1c77fad24351469db34e71b369c423ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:26.968109 30641 consensus_queue.cc:1035] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:26.968604 30607 consensus_queue.cc:1035] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:26.977753 30603 tablet_replica.cc:439] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8: TxnStatusTablet state changed. Reason: New leader 1c77fad24351469db34e71b369c423ac. Latest consensus state: current_term: 1 leader_uuid: "1c77fad24351469db34e71b369c423ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } }
I20260923 17:21:26.978068 30603 tablet_replica.cc:442] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:21:26.978753 30602 tablet_replica.cc:439] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb: TxnStatusTablet state changed. Reason: New leader 1c77fad24351469db34e71b369c423ac. Latest consensus state: current_term: 1 leader_uuid: "1c77fad24351469db34e71b369c423ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } }
I20260923 17:21:26.979189 30602 tablet_replica.cc:442] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:21:26.983161 30604 tablet_replica.cc:439] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c77fad24351469db34e71b369c423ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } }
I20260923 17:21:26.983556 30604 tablet_replica.cc:442] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:21:26.983282 30641 tablet_replica.cc:439] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c77fad24351469db34e71b369c423ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } }
I20260923 17:21:26.983754 30641 tablet_replica.cc:442] T 6ba92752854e47ccb0cf61ac9448a42a P 1c77fad24351469db34e71b369c423ac: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:21:26.986374 30603 tablet_replica.cc:439] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c77fad24351469db34e71b369c423ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } }
I20260923 17:21:26.986650 30603 tablet_replica.cc:442] T 6ba92752854e47ccb0cf61ac9448a42a P d25b5fb40d2d4c8b8bb4145e0ee482f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:21:26.986846 30643 mvcc.cc:205] Tried to move back new op lower bound from 7332594020211130368 to 7332594019859345408. Current Snapshot: MvccSnapshot[applied={T|T < 7332594020211130368}]
I20260923 17:21:26.988979 30640 mvcc.cc:205] Tried to move back new op lower bound from 7332594020211130368 to 7332594019859345408. Current Snapshot: MvccSnapshot[applied={T|T < 7332594020211130368 or (T in {7332594020211130368})}]
I20260923 17:21:26.990978 30602 tablet_replica.cc:439] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c77fad24351469db34e71b369c423ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d25b5fb40d2d4c8b8bb4145e0ee482f8" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 45615 } } peers { permanent_uuid: "1c77fad24351469db34e71b369c423ac" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38479 } } peers { permanent_uuid: "29a52b5aff154351afd4ae713fb8f6eb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 45473 } } }
I20260923 17:21:26.991310 30602 tablet_replica.cc:442] T 6ba92752854e47ccb0cf61ac9448a42a P 29a52b5aff154351afd4ae713fb8f6eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:21:27.009868 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:27.003089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:42219 (local address 127.28.72.194:38479)
0923 17:21:27.003333 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:27.003342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:27.004038 (+ 696us) server_negotiation.cc:407] Connection header received
0923 17:21:27.004306 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:27.004313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:27.004385 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:27.004480 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:27.005061 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:27.005874 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:27.007942 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:27.008388 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:27.008545 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:27.009027 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:27.009139 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:27.009270 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:27.009413 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 17:21:27.009483 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 17:21:27.009944 30653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:27.003107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:38479 (local address 127.28.72.194:42219)
0923 17:21:27.003919 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:27.003942 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:27.004157 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:27.004501 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:27.004504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:27.004515 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:21:27.004880 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:27.004890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:27.006026 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:27.006033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:27.007759 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:27.007774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:27.008595 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:27.008603 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:27.008739 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:27.008802 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:21:27.009278 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:21:27.009290 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 17:21:27.009388 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 17:21:27.009529 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":746,"threads_started":1}
I20260923 17:21:27.092214 30091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:27.092504 30091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:27.096653 30091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81b17736261949febc2c8ad1385b5b38: Sending DeleteTablet for 3 replicas of tablet 830f22814f004b6984d596b72320849c
I20260923 17:21:27.097819 30228 tablet_service.cc:1514] Processing DeleteTablet for tablet 830f22814f004b6984d596b72320849c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:27 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:59528
I20260923 17:21:27.097824 30367 tablet_service.cc:1514] Processing DeleteTablet for tablet 830f22814f004b6984d596b72320849c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:27 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:60180
I20260923 17:21:27.098398 30507 tablet_service.cc:1514] Processing DeleteTablet for tablet 830f22814f004b6984d596b72320849c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:27 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:59278
I20260923 17:21:27.113088 30655 tablet_replica.cc:330] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8: stopping tablet replica
I20260923 17:21:27.113386 30655 raft_consensus.cc:2229] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:27.113684 30655 raft_consensus.cc:2258] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:27.115368 30655 ts_tablet_manager.cc:1905] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:27.117898 30656 tablet_replica.cc:330] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb: stopping tablet replica
I20260923 17:21:27.118465 30658 tablet_replica.cc:330] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac: stopping tablet replica
I20260923 17:21:27.119202 30658 raft_consensus.cc:2229] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:21:27.119370 30656 raft_consensus.cc:2229] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:27.119661 30656 raft_consensus.cc:2258] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:27.120110 30658 raft_consensus.cc:2258] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:27.121920 30656 ts_tablet_manager.cc:1905] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:27.123224 30658 ts_tablet_manager.cc:1905] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:27.124547 30655 ts_tablet_manager.cc:1918] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:21:27.124697 30655 log.cc:1196] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/830f22814f004b6984d596b72320849c
I20260923 17:21:27.125177 30655 ts_tablet_manager.cc:1939] T 830f22814f004b6984d596b72320849c P d25b5fb40d2d4c8b8bb4145e0ee482f8: Deleting consensus metadata
I20260923 17:21:27.126473 30078 catalog_manager.cc:4841] TS d25b5fb40d2d4c8b8bb4145e0ee482f8 (127.28.72.193:45615): tablet 830f22814f004b6984d596b72320849c (table test-table [id=ee0023745b31438e8b809f246c236c11]) successfully deleted
I20260923 17:21:27.132997 30658 ts_tablet_manager.cc:1918] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:21:27.132997 30656 ts_tablet_manager.cc:1918] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:21:27.133170 30658 log.cc:1196] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/830f22814f004b6984d596b72320849c
I20260923 17:21:27.133361 30656 log.cc:1196] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/830f22814f004b6984d596b72320849c
I20260923 17:21:27.134200 30658 ts_tablet_manager.cc:1939] T 830f22814f004b6984d596b72320849c P 1c77fad24351469db34e71b369c423ac: Deleting consensus metadata
I20260923 17:21:27.134356 30656 ts_tablet_manager.cc:1939] T 830f22814f004b6984d596b72320849c P 29a52b5aff154351afd4ae713fb8f6eb: Deleting consensus metadata
I20260923 17:21:27.135520 30077 catalog_manager.cc:4841] TS 29a52b5aff154351afd4ae713fb8f6eb (127.28.72.195:45473): tablet 830f22814f004b6984d596b72320849c (table test-table [id=ee0023745b31438e8b809f246c236c11]) successfully deleted
I20260923 17:21:27.135601 30075 catalog_manager.cc:4841] TS 1c77fad24351469db34e71b369c423ac (127.28.72.194:38479): tablet 830f22814f004b6984d596b72320849c (table test-table [id=ee0023745b31438e8b809f246c236c11]) successfully deleted
Sep 23 17:21:27 dist-test-slave-j0m4 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184086, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:21:27.148847 30659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:27.112158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.28.72.193:45615)
0923 17:21:27.130532 (+ 18374us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:27.130538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:27.130565 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:21:27.130637 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:27.130642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:27.130738 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:27.130878 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:27.133313 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:27.134530 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:27.136635 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:27.137118 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:27.137260 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:27.142195 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:27.142212 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:27.142230 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:27.142271 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:27.145615 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:27.146586 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:27.146595 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:27.146597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:27.146743 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:27.147217 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:27.147221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:27.147223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:27.147598 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:27.147906 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:27.148245 (+ 339us) server_negotiation.cc:299] Negotiation successful
0923 17:21:27.148496 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18190,"thread_start_us":20923,"threads_started":1}
W20260923 17:21:27.149592 30293 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59534
I20260923 17:21:27.167354 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:27.152905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.28.72.254:45335)
0923 17:21:27.153617 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:27.153624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:27.153645 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:21:27.153708 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:27.153713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:27.153790 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:27.153919 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:27.155320 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:27.156499 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:27.157730 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:27.158388 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:27.158520 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:27.160871 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:27.160878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:27.160883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:27.160932 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:27.164297 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:27.165209 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:27.165215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:27.165218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:27.165367 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:27.165757 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:27.165762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:27.165765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:27.166136 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:27.166355 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:27.166693 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 17:21:27.166928 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":448,"threads_started":1}
W20260923 17:21:27.168072 30091 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34534
I20260923 17:21:27.170854 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 30162
I20260923 17:21:27.186313 30659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:27.171976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.28.72.193:45615)
0923 17:21:27.172182 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:27.172189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:27.172464 (+ 275us) server_negotiation.cc:407] Connection header received
0923 17:21:27.172610 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:27.172615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:27.172708 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:27.172813 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:27.174283 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:27.175394 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:27.176803 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:27.177271 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:27.177437 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:27.179939 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:27.179944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:27.179947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:27.179992 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:27.182859 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:27.183650 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:27.183657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:27.183661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:27.183837 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:27.184266 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:27.184271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:27.184274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:27.184666 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:27.184965 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:27.185896 (+ 931us) server_negotiation.cc:299] Negotiation successful
0923 17:21:27.186074 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260923 17:21:27.187317 30293 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59548
W20260923 17:21:27.188038 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 30162: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:21:27.188285 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 30162
I20260923 17:21:27.213174 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 30302
Sep 23 17:21:27 dist-test-slave-j0m4 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184086, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:21:27.234208 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:27.214087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60210 (local address 127.28.72.194:38479)
0923 17:21:27.214296 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:27.214300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:27.214318 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:21:27.214561 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:27.214566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:27.214662 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:27.214752 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:27.216481 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:27.217327 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:27.219476 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:27.220203 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:27.220320 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:27.226250 (+ 5930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:27.226271 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:27.226292 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:27.226342 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:27.230696 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:27.231700 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:27.231710 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:27.231713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:27.231925 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:27.232369 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:27.232375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:27.232378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:27.232855 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:27.233196 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:27.233522 (+ 326us) server_negotiation.cc:299] Negotiation successful
0923 17:21:27.233769 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260923 17:21:27.235069 30432 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60210
W20260923 17:21:27.235900 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 30302: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:21:27.236088 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 30302
I20260923 17:21:27.260205 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 30441
Sep 23 17:21:27 dist-test-slave-j0m4 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184086, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:21:27.277045 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:27.260923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59294 (local address 127.28.72.195:45473)
0923 17:21:27.261543 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:27.261548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:27.261565 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:21:27.261617 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:27.261621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:27.261682 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:27.261783 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:27.262948 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:27.263846 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:27.265081 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:27.265691 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:27.265801 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:27.270400 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:27.270417 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:27.270434 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:27.270481 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:27.273940 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:27.274788 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:27.274796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:27.274799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:27.274943 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:27.275377 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:27.275382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:27.275386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:27.275754 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:27.276087 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:27.276403 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 17:21:27.276658 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":293,"threads_started":1}
W20260923 17:21:27.277822 30572 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59294
W20260923 17:21:27.278555 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 30441: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:21:27.278738 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 30441
I20260923 17:21:27.296056 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 30059
I20260923 17:21:27.297279 30131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:27.413636 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 30059
2026-09-23T17:21:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4959 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:21:27 dist-test-slave-j0m4 krb5kdc[30669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:21:27 dist-test-slave-j0m4 krb5kdc[30669](info): set up 2 sockets
Sep 23 17:21:27 dist-test-slave-j0m4 krb5kdc[30669](info): commencing operation
krb5kdc: starting...
W20260923 17:21:28.543598 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:21:28 dist-test-slave-j0m4 krb5kdc[30669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:21:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:21:28.764446 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:44581
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:44581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:29.134614 30682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:29.134940 30682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:29.135128 30682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:29.142737 30682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:29.142874 30682 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:21:29.142916 30682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:29.142971 30682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:29.143013 30682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:29.143077 30682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:21:29.153764 30682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:44581
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:44581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:29.155321 30682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:29.156739 30682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:29.170120 30687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:29.170172 30690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:29.170078 30688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:29.171341 30682 server_base.cc:1034] running on GCE node
I20260923 17:21:29.172183 30682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:29.173702 30682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:29.174991 30682 hybrid_clock.cc:648] HybridClock initialized: now 1790184089174921 us; error 30 us; skew 500 ppm
I20260923 17:21:29.175477 30682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:29 dist-test-slave-j0m4 krb5kdc[30669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184089, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:29.179201 30682 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:21:29.180893 30682 webserver.cc:460] Webserver started at http://127.28.72.254:42579/ using document root <none> and password file <none>
I20260923 17:21:29.181751 30682 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:29.181856 30682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:29.182199 30682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:29.185561 30682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "d1dcec500acf4607ba8d68d106a27a8c"
format_stamp: "Formatted at 2026-09-23 17:21:29 on dist-test-slave-j0m4"
server_key: "7d6ae2767b2c65525e2ae6a5f2e2f4d5"
server_key_iv: "1dbe80628b2f2c05aefbd2efcec27688"
server_key_version: "encryptionkey@0"
I20260923 17:21:29.186550 30682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "d1dcec500acf4607ba8d68d106a27a8c"
format_stamp: "Formatted at 2026-09-23 17:21:29 on dist-test-slave-j0m4"
server_key: "7d6ae2767b2c65525e2ae6a5f2e2f4d5"
server_key_iv: "1dbe80628b2f2c05aefbd2efcec27688"
server_key_version: "encryptionkey@0"
I20260923 17:21:29.192812 30682 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260923 17:21:29.197321 30697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:29.199218 30682 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:21:29.199448 30682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "d1dcec500acf4607ba8d68d106a27a8c"
format_stamp: "Formatted at 2026-09-23 17:21:29 on dist-test-slave-j0m4"
server_key: "7d6ae2767b2c65525e2ae6a5f2e2f4d5"
server_key_iv: "1dbe80628b2f2c05aefbd2efcec27688"
server_key_version: "encryptionkey@0"
I20260923 17:21:29.199697 30682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:29.227187 30682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:29.232105 30682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:29.232424 30682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:29.256505 30682 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:44581
I20260923 17:21:29.256538 30758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:44581 every 8 connection(s)
I20260923 17:21:29.257937 30682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:29.262601 30759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:29.266812 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 30682
I20260923 17:21:29.267089 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:21:29.268040 28963 external_mini_cluster.cc:1428] Setting key 5740c85c51064f787400cc8fd8c8deff
I20260923 17:21:29.277462 30759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c: Bootstrap starting.
I20260923 17:21:29.281338 30759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:29.282565 30759 log.cc:824] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:21:29 dist-test-slave-j0m4 krb5kdc[30669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:29.286109 30759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c: No bootstrap required, opened a new log
I20260923 17:21:29.293287 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:29.270546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33096 (local address 127.28.72.254:44581)
0923 17:21:29.271571 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:29.271586 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:29.271632 (+ 46us) server_negotiation.cc:407] Connection header received
0923 17:21:29.273425 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:29.273452 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:29.273964 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:29.274477 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:29.276058 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:29.277452 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:29.278876 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:29.279725 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:29.279949 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:29.284375 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:29.284384 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:29.284403 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:29.284446 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:29.288059 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:29.288983 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:29.289004 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:29.289012 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:29.289241 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:29.289705 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:29.289710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:29.289714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:29.290389 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:29.290755 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:29.291358 (+ 603us) server_negotiation.cc:299] Negotiation successful
0923 17:21:29.292057 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":440,"threads_started":1}
I20260923 17:21:29.296023 30759 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1dcec500acf4607ba8d68d106a27a8c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44581 } }
I20260923 17:21:29.296501 30759 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:29.296650 30759 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1dcec500acf4607ba8d68d106a27a8c, State: Initialized, Role: FOLLOWER
I20260923 17:21:29.297483 30759 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1dcec500acf4607ba8d68d106a27a8c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44581 } }
I20260923 17:21:29.297729 30759 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:29.297890 30759 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:29.298197 30759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:29.301381 30759 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1dcec500acf4607ba8d68d106a27a8c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44581 } }
I20260923 17:21:29.302052 30759 leader_election.cc:302] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1dcec500acf4607ba8d68d106a27a8c; no voters:
I20260923 17:21:29.302732 30759 leader_election.cc:288] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:29.302922 30764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:29.303825 30764 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [term 1 LEADER]: Becoming Leader. State: Replica: d1dcec500acf4607ba8d68d106a27a8c, State: Running, Role: LEADER
I20260923 17:21:29.304490 30764 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1dcec500acf4607ba8d68d106a27a8c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44581 } }
I20260923 17:21:29.305289 30759 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:29.307583 30766 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1dcec500acf4607ba8d68d106a27a8c. Latest consensus state: current_term: 1 leader_uuid: "d1dcec500acf4607ba8d68d106a27a8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1dcec500acf4607ba8d68d106a27a8c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44581 } } }
I20260923 17:21:29.308175 30765 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1dcec500acf4607ba8d68d106a27a8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1dcec500acf4607ba8d68d106a27a8c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44581 } } }
I20260923 17:21:29.308476 30765 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:29.308185 30766 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:29.310179 30772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:29.316680 30772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:29.328163 30772 catalog_manager.cc:1347] Generated new cluster ID: 11fe70f0a6c441b48970eef1ba41fb2e
I20260923 17:21:29.328301 30772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:29.341835 30772 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:29.343504 30772 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:29.357602 30772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c: Generated new TSK 0
I20260923 17:21:29.358615 30772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:21:29.415217 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:44581
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:29.811201 30785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:29.811563 30785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:29.811739 30785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:29.819505 30785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:29.819671 30785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:29.819824 30785 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:21:29.819911 30785 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:21:29.820045 30785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:21:29.830377 30785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.72.254:44581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:29.831914 30785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:29.833285 30785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:29.849344 30791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:29.849370 30793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:29.850083 30785 server_base.cc:1034] running on GCE node
W20260923 17:21:29.850162 30790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:29.850891 30785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:29.852056 30785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:29.853300 30785 hybrid_clock.cc:648] HybridClock initialized: now 1790184089853246 us; error 41 us; skew 500 ppm
I20260923 17:21:29.853696 30785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:29 dist-test-slave-j0m4 krb5kdc[30669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184089, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:29.857278 30785 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:21:29.859223 30785 webserver.cc:460] Webserver started at http://127.28.72.193:46571/ using document root <none> and password file <none>
I20260923 17:21:29.860028 30785 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:29.860176 30785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:29.860541 30785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:29.863865 30785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "50971779fba844e9b00cb0605df42e35"
format_stamp: "Formatted at 2026-09-23 17:21:29 on dist-test-slave-j0m4"
server_key: "e84b2341aaecb534464ca18ae6118ea0"
server_key_iv: "df675175ee4f02139973996f4e3a2988"
server_key_version: "encryptionkey@0"
I20260923 17:21:29.864681 30785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "50971779fba844e9b00cb0605df42e35"
format_stamp: "Formatted at 2026-09-23 17:21:29 on dist-test-slave-j0m4"
server_key: "e84b2341aaecb534464ca18ae6118ea0"
server_key_iv: "df675175ee4f02139973996f4e3a2988"
server_key_version: "encryptionkey@0"
I20260923 17:21:29.870559 30785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 17:21:29.874862 30800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:29.876657 30785 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:21:29.876900 30785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "50971779fba844e9b00cb0605df42e35"
format_stamp: "Formatted at 2026-09-23 17:21:29 on dist-test-slave-j0m4"
server_key: "e84b2341aaecb534464ca18ae6118ea0"
server_key_iv: "df675175ee4f02139973996f4e3a2988"
server_key_version: "encryptionkey@0"
I20260923 17:21:29.877159 30785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:29.896725 30785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:29.901420 30785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:29.901742 30785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:29.904810 30785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:29.904956 30785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:29.905087 30785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:29.905195 30785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:21:29 dist-test-slave-j0m4 krb5kdc[30669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184089, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:29.935256 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:29.905409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:55299 (local address 127.28.72.254:44581)
0923 17:21:29.908032 (+ 2623us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:29.908039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:29.908092 (+ 53us) server_negotiation.cc:407] Connection header received
0923 17:21:29.910250 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:29.910260 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:29.910375 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:29.910531 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:29.913437 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:29.914827 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:29.917934 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:29.918727 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:29.918905 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:29.925573 (+ 6668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:29.925584 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:29.925592 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:29.925640 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:29.929914 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:29.931470 (+ 1556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:29.931479 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:29.931485 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:29.931796 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:29.932552 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:29.932558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:29.932562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:29.932874 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:29.933002 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:29.934389 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0923 17:21:29.934698 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2473,"thread_start_us":2324,"threads_started":1}
I20260923 17:21:29.937245 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:29.906084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:44581 (local address 127.28.72.193:55299)
0923 17:21:29.907310 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:29.907396 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:29.909773 (+ 2377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:29.910799 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:29.910812 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:29.911888 (+ 1076us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:29.913167 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:29.913215 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:29.915086 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:29.915095 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:29.917706 (+ 2611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:29.917734 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:29.919161 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:29.919177 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:29.919782 (+ 605us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:29.920812 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:29.920869 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:29.925157 (+ 4288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:29.930183 (+ 5026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:29.930203 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:29.930232 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:29.931265 (+ 1033us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:29.931990 (+ 725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:29.931998 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:29.932002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:29.932383 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:29.933024 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:29.933037 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:29.933727 (+ 690us) client_negotiation.cc:769] Sending connection context
0923 17:21:29.934191 (+ 464us) client_negotiation.cc:240] Negotiation successful
0923 17:21:29.935006 (+ 815us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":448,"threads_started":1}
I20260923 17:21:29.952013 30785 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:41835
I20260923 17:21:29.952066 30920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:41835 every 8 connection(s)
I20260923 17:21:29.953644 30785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:21:29.956332 30921 heartbeater.cc:347] Connected to a master server at 127.28.72.254:44581
I20260923 17:21:29.956769 30921 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:29.957703 30921 heartbeater.cc:511] Master 127.28.72.254:44581 requested a full tablet report, sending...
I20260923 17:21:29.960275 30714 ts_manager.cc:194] Registered new tserver with Master: 50971779fba844e9b00cb0605df42e35 (127.28.72.193:41835)
I20260923 17:21:29.960336 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 30785
I20260923 17:21:29.960592 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:21:29.961419 28963 external_mini_cluster.cc:1428] Setting key c261096b80c69f1e6c668ba0cc3ba48a
I20260923 17:21:29.962306 30714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:55299
I20260923 17:21:29.963610 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:29.946263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:52145 (local address 127.28.72.254:44581)
0923 17:21:29.946530 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:29.946541 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:29.947779 (+ 1238us) server_negotiation.cc:407] Connection header received
0923 17:21:29.948024 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:29.948031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:29.948140 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:29.948280 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:29.950116 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:29.951253 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:29.953416 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:29.953953 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:29.954073 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:29.956481 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:29.956491 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:29.956499 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:29.956544 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:29.960114 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:29.961079 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:29.961087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:29.961092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:29.961300 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:29.961764 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:29.961769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:29.961772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:29.962209 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:29.962397 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:29.962923 (+ 526us) server_negotiation.cc:299] Negotiation successful
0923 17:21:29.963239 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":86}
I20260923 17:21:29.963722 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:29.946194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:44581 (local address 127.28.72.193:52145)
0923 17:21:29.947611 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:29.947637 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:29.947854 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:29.948334 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:29.948342 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:29.949338 (+ 996us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:29.949894 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:29.949909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:29.951520 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:29.951525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:29.953201 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:29.953216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:29.954181 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:29.954191 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:29.954532 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:29.955395 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:29.955422 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:29.956251 (+ 829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:29.960283 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:29.960289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:29.960296 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:29.960924 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:29.961455 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:29.961461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:29.961465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:29.961622 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:29.962366 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:29.962372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:29.962608 (+ 236us) client_negotiation.cc:769] Sending connection context
0923 17:21:29.962799 (+ 191us) client_negotiation.cc:240] Negotiation successful
0923 17:21:29.963199 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"thread_start_us":931,"threads_started":1}
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:21:30.011430 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:44581
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:30.350663 30925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:30.351033 30925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:30.351215 30925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:30.358932 30925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:30.359151 30925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:30.359321 30925 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:21:30.359412 30925 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:21:30.359536 30925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:21:30.370100 30925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.72.254:44581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:30.371606 30925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:30.373268 30925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:30.387578 30931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:30.387614 30933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:30.388551 30925 server_base.cc:1034] running on GCE node
W20260923 17:21:30.388700 30930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:30.389436 30925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:30.390539 30925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:30.391813 30925 hybrid_clock.cc:648] HybridClock initialized: now 1790184090391765 us; error 40 us; skew 500 ppm
I20260923 17:21:30.392198 30925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:30 dist-test-slave-j0m4 krb5kdc[30669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184090, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:30.401777 30925 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:21:30.403503 30925 webserver.cc:460] Webserver started at http://127.28.72.194:34063/ using document root <none> and password file <none>
I20260923 17:21:30.404305 30925 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:30.404423 30925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:30.404734 30925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:30.408043 30925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "70b4fccd75ba448291d4ce8bc00a6653"
format_stamp: "Formatted at 2026-09-23 17:21:30 on dist-test-slave-j0m4"
server_key: "1980f5117245d5f73cd707b6707e3d0b"
server_key_iv: "9f49a1e8b4eec5ce27d3d74bf239c595"
server_key_version: "encryptionkey@0"
I20260923 17:21:30.408775 30925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "70b4fccd75ba448291d4ce8bc00a6653"
format_stamp: "Formatted at 2026-09-23 17:21:30 on dist-test-slave-j0m4"
server_key: "1980f5117245d5f73cd707b6707e3d0b"
server_key_iv: "9f49a1e8b4eec5ce27d3d74bf239c595"
server_key_version: "encryptionkey@0"
I20260923 17:21:30.414347 30925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:21:30.418263 30940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:30.419982 30925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:21:30.420138 30925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "70b4fccd75ba448291d4ce8bc00a6653"
format_stamp: "Formatted at 2026-09-23 17:21:30 on dist-test-slave-j0m4"
server_key: "1980f5117245d5f73cd707b6707e3d0b"
server_key_iv: "9f49a1e8b4eec5ce27d3d74bf239c595"
server_key_version: "encryptionkey@0"
I20260923 17:21:30.420344 30925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:30.440449 30925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:30.445201 30925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:30.445521 30925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:30.448467 30925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:30.448568 30925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:30.448724 30925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:30.448801 30925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:21:30 dist-test-slave-j0m4 krb5kdc[30669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184090, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:30.477203 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:30.449072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:48285 (local address 127.28.72.254:44581)
0923 17:21:30.449264 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:30.449269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:30.450929 (+ 1660us) server_negotiation.cc:407] Connection header received
0923 17:21:30.454060 (+ 3131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:30.454070 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:30.454184 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:30.454297 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:30.457613 (+ 3316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:30.458654 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:30.462147 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:30.462760 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:30.462932 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:30.469241 (+ 6309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:30.469249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:30.469256 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:30.469312 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:30.472926 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:30.474249 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:30.474254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:30.474257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:30.474398 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:30.475224 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:30.475232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:30.475237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:30.475644 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:30.475823 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:30.476623 (+ 800us) server_negotiation.cc:299] Negotiation successful
0923 17:21:30.476855 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:21:30.478302 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:30.449738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:44581 (local address 127.28.72.194:48285)
0923 17:21:30.450721 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:30.450781 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:30.453618 (+ 2837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:30.454968 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:30.454986 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:30.456068 (+ 1082us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:30.457350 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:30.457391 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:30.458880 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:30.458889 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:30.461896 (+ 3007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:30.461918 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:30.463143 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:30.463154 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:30.463588 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:30.464643 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:30.464693 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:30.468839 (+ 4146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:30.473149 (+ 4310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:30.473166 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:30.473191 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:30.474090 (+ 899us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:30.474625 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:30.474631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:30.474634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:30.475030 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:30.475793 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:30.475802 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:30.476223 (+ 421us) client_negotiation.cc:769] Sending connection context
0923 17:21:30.476512 (+ 289us) client_negotiation.cc:240] Negotiation successful
0923 17:21:30.477087 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":446,"threads_started":1}
I20260923 17:21:30.495635 30925 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:34693
I20260923 17:21:30.495672 31059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:34693 every 8 connection(s)
I20260923 17:21:30.497097 30925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:21:30.499563 31060 heartbeater.cc:347] Connected to a master server at 127.28.72.254:44581
I20260923 17:21:30.499794 31060 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:30.500491 31060 heartbeater.cc:511] Master 127.28.72.254:44581 requested a full tablet report, sending...
I20260923 17:21:30.501497 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 30925
I20260923 17:21:30.501711 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:21:30.501991 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:30.486393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:44657 (local address 127.28.72.254:44581)
0923 17:21:30.486625 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:30.486632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:30.487397 (+ 765us) server_negotiation.cc:407] Connection header received
0923 17:21:30.487610 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:30.487618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:30.487699 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:30.487824 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:30.489264 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:30.490224 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:30.492275 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:30.492841 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:30.492972 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:30.495526 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:30.495534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:30.495543 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:30.495572 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:30.498596 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:30.499612 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:30.499617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:30.499619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:30.499798 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:30.500278 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:30.500284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:30.500287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:30.500728 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:30.500896 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:30.501364 (+ 468us) server_negotiation.cc:299] Negotiation successful
0923 17:21:30.501625 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:21:30.502044 31035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:30.486479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:44581 (local address 127.28.72.194:44657)
0923 17:21:30.487247 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:30.487274 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:30.487465 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:30.487860 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:30.487864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:30.488541 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:30.489032 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:30.489055 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:30.490415 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:30.490422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:30.492050 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:30.492062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:30.493231 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:30.493241 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:30.493476 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:30.494190 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:30.494207 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:30.495322 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:30.498736 (+ 3414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:30.498740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:30.498743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:30.499332 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:30.499967 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:30.499972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:30.499976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:30.500150 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:30.500850 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:30.500856 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:30.501049 (+ 193us) client_negotiation.cc:769] Sending connection context
0923 17:21:30.501222 (+ 173us) client_negotiation.cc:240] Negotiation successful
0923 17:21:30.501580 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":342,"threads_started":1}
I20260923 17:21:30.502089 30714 ts_manager.cc:194] Registered new tserver with Master: 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194:34693)
I20260923 17:21:30.502255 28963 external_mini_cluster.cc:1428] Setting key 33aadf3b586fffdd16fd2d9c5a541721
I20260923 17:21:30.502985 30714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:48285
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:21:30.547458 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:44581
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:30.896418 31064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:30.896728 31064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:30.896894 31064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:30.904632 31064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:30.904753 31064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:30.904948 31064 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:21:30.905078 31064 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:21:30.905200 31064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:21:30.915409 31064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.72.254:44581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:30.916728 31064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:30.918056 31064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:30.932729 31069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:30.932827 31070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:30.933240 31072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:30.933578 31064 server_base.cc:1034] running on GCE node
I20260923 17:21:30.934240 31064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:30.935387 31064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:30.936643 31064 hybrid_clock.cc:648] HybridClock initialized: now 1790184090936574 us; error 63 us; skew 500 ppm
I20260923 17:21:30.937016 31064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:30 dist-test-slave-j0m4 krb5kdc[30669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184090, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:30.940603 31064 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:21:30.942255 31064 webserver.cc:460] Webserver started at http://127.28.72.195:39565/ using document root <none> and password file <none>
I20260923 17:21:30.942986 31064 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:30.943184 31064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:30.943526 31064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:30.946630 31064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "cae72425ef074680b605c96678ced123"
format_stamp: "Formatted at 2026-09-23 17:21:30 on dist-test-slave-j0m4"
server_key: "e5cd85352cbd9f10f6b21dfaca42b35f"
server_key_iv: "2e7d5cffe821f96450f3b200cd43923a"
server_key_version: "encryptionkey@0"
I20260923 17:21:30.947520 31064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "cae72425ef074680b605c96678ced123"
format_stamp: "Formatted at 2026-09-23 17:21:30 on dist-test-slave-j0m4"
server_key: "e5cd85352cbd9f10f6b21dfaca42b35f"
server_key_iv: "2e7d5cffe821f96450f3b200cd43923a"
server_key_version: "encryptionkey@0"
I20260923 17:21:30.953138 31064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:21:30.957173 31079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:30.959203 31064 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 17:21:30.959425 31064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "cae72425ef074680b605c96678ced123"
format_stamp: "Formatted at 2026-09-23 17:21:30 on dist-test-slave-j0m4"
server_key: "e5cd85352cbd9f10f6b21dfaca42b35f"
server_key_iv: "2e7d5cffe821f96450f3b200cd43923a"
server_key_version: "encryptionkey@0"
I20260923 17:21:30.959661 31064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:30.967036 30921 heartbeater.cc:503] Master 127.28.72.254:44581 was elected leader, sending a full tablet report...
I20260923 17:21:31.002869 31064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:31.008275 31064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:31.008611 31064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:31.011850 31064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:31.012046 31064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:31.012188 31064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:31.012282 31064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:21:31 dist-test-slave-j0m4 krb5kdc[30669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184090, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:31.039505 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.013027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:41497 (local address 127.28.72.254:44581)
0923 17:21:31.013570 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:31.013578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:31.015170 (+ 1592us) server_negotiation.cc:407] Connection header received
0923 17:21:31.018350 (+ 3180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:31.018359 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:31.018473 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:31.018631 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:31.020608 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.022022 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.025463 (+ 3441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.026274 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.026447 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.030981 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:31.030992 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:31.030999 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:31.031056 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:31.035309 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:31.036653 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:31.036663 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:31.036667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:31.036845 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:31.037390 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:31.037395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:31.037399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:31.037715 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:31.037892 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:31.038775 (+ 883us) server_negotiation.cc:299] Negotiation successful
0923 17:21:31.039033 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":247,"threads_started":1}
I20260923 17:21:31.040977 31089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.013807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:44581 (local address 127.28.72.195:41497)
0923 17:21:31.014911 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:31.014972 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:31.017928 (+ 2956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:31.019007 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:31.019016 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:31.019695 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:31.020388 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.020408 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.022270 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.022279 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.025255 (+ 2976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.025272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.026663 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.026670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.026906 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.027767 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:31.027794 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:31.030616 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:31.035514 (+ 4898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:31.035529 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:31.035558 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:31.036456 (+ 898us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:31.036986 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:31.036992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:31.036996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:31.037244 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:31.037900 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:31.037913 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:31.038342 (+ 429us) client_negotiation.cc:769] Sending connection context
0923 17:21:31.038657 (+ 315us) client_negotiation.cc:240] Negotiation successful
0923 17:21:31.039332 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":476,"threads_started":1}
I20260923 17:21:31.069283 31064 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:36001
I20260923 17:21:31.069337 31199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:36001 every 8 connection(s)
I20260923 17:21:31.070780 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.051953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:44581 (local address 127.28.72.195:52739)
0923 17:21:31.052709 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:31.052736 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:31.052927 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:31.053380 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:31.053383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:31.053853 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:31.054152 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.054161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.056000 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.056012 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.058378 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.058404 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.059721 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.059727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.059864 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.060758 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:31.060787 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:31.062687 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:31.067579 (+ 4892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:31.067585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:31.067589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:31.068189 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:31.068907 (+ 718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:31.068913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:31.068916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:31.069120 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:31.069801 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:31.069804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:31.069979 (+ 175us) client_negotiation.cc:769] Sending connection context
0923 17:21:31.070127 (+ 148us) client_negotiation.cc:240] Negotiation successful
0923 17:21:31.070434 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":439,"threads_started":1}
I20260923 17:21:31.071018 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.052584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:52739 (local address 127.28.72.254:44581)
0923 17:21:31.052816 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:31.052822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:31.052849 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:21:31.053122 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:31.053133 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:31.053234 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:31.053360 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:31.054335 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.055805 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.058626 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.059409 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.059596 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.063041 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:31.063053 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:31.063061 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:31.063153 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:31.067398 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:31.068508 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:31.068515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:31.068518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:31.068721 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:31.069277 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:31.069282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:31.069285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:31.069662 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:31.069817 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:31.070296 (+ 479us) server_negotiation.cc:299] Negotiation successful
0923 17:21:31.070676 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 17:21:31.071267 31064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:21:31.073679 31200 heartbeater.cc:347] Connected to a master server at 127.28.72.254:44581
I20260923 17:21:31.074005 31200 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:31.074761 31200 heartbeater.cc:511] Master 127.28.72.254:44581 requested a full tablet report, sending...
I20260923 17:21:31.076297 30713 ts_manager.cc:194] Registered new tserver with Master: cae72425ef074680b605c96678ced123 (127.28.72.195:36001)
I20260923 17:21:31.077188 30713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:41497
I20260923 17:21:31.078972 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 31064
I20260923 17:21:31.079217 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:21:31.079806 28963 external_mini_cluster.cc:1428] Setting key cfe7af1f0697b53adc9837d0e0689975
I20260923 17:21:31.083707 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:21:31 dist-test-slave-j0m4 krb5kdc[30669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:21:31 dist-test-slave-j0m4 krb5kdc[30669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:31.137533 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.120205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.28.72.254:44581)
0923 17:21:31.121808 (+ 1603us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:31.121812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:31.122380 (+ 568us) server_negotiation.cc:407] Connection header received
0923 17:21:31.122576 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:31.122581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:31.122649 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:31.122742 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:31.124995 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.125766 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.127202 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.127675 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.127873 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.131858 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:31.131863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:31.131867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:31.131933 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:31.134710 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:31.135461 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:31.135465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:31.135468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:31.135587 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:31.136001 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:31.136006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:31.136008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:31.136311 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:31.136533 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:31.136856 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 17:21:31.137043 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1429}
I20260923 17:21:31.142077 30713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:21:31.150137 30713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:21:31.177135 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.166553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.28.72.194:34693)
0923 17:21:31.168452 (+ 1899us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:31.168460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:31.169011 (+ 551us) server_negotiation.cc:407] Connection header received
0923 17:21:31.169173 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:31.169183 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:31.169444 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:31.169668 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:31.170189 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.171544 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.173690 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.175287 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.175440 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.175685 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:31.176684 (+ 999us) server_negotiation.cc:299] Negotiation successful
0923 17:21:31.176841 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1708,"thread_start_us":1586,"threads_started":1}
I20260923 17:21:31.177150 31214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.164039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.28.72.195:36001)
0923 17:21:31.166452 (+ 2413us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:31.166461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:31.166495 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:21:31.167234 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:31.167245 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:31.167557 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:31.167757 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:31.168714 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.170453 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.173928 (+ 3475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.175427 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.175550 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.175748 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:31.176592 (+ 844us) server_negotiation.cc:299] Negotiation successful
0923 17:21:31.176781 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2177,"thread_start_us":1957,"threads_started":1}
I20260923 17:21:31.177142 31216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.164996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.28.72.193:41835)
0923 17:21:31.166690 (+ 1694us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:31.166700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:31.167581 (+ 881us) server_negotiation.cc:407] Connection header received
0923 17:21:31.167702 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:31.167713 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:31.167992 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:31.168667 (+ 675us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:31.169494 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.170917 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.174053 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.175495 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.175657 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.175863 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:31.176598 (+ 735us) server_negotiation.cc:299] Negotiation successful
0923 17:21:31.176789 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1533,"thread_start_us":1389,"threads_started":1}
I20260923 17:21:31.177204 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.164547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:41835 (local address 127.0.0.1:58240)
0923 17:21:31.167236 (+ 2689us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:31.167265 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:31.167422 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:31.168807 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:31.168812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:31.168837 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:31.169306 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.169317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.171098 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.171103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.173862 (+ 2759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.173879 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.176033 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.176044 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.176335 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.176384 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 17:21:31.176527 (+ 143us) client_negotiation.cc:240] Negotiation successful
0923 17:21:31.176767 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2464,"mutex_wait_us":73,"thread_start_us":2312,"threads_started":1}
I20260923 17:21:31.178072 31217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.166074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:34693 (local address 127.0.0.1:50538)
0923 17:21:31.168867 (+ 2793us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:31.168895 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:31.169026 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:31.169651 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:31.169655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:31.169680 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:31.170029 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.170038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.171712 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.171724 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.173501 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.173522 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.175839 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.175847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.176222 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.176349 (+ 127us) client_negotiation.cc:769] Sending connection context
0923 17:21:31.176503 (+ 154us) client_negotiation.cc:240] Negotiation successful
0923 17:21:31.177583 (+ 1080us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2666,"mutex_wait_us":960,"thread_start_us":2533,"threads_started":1}
I20260923 17:21:31.178124 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.163880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36001 (local address 127.0.0.1:46006)
0923 17:21:31.165369 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:31.165430 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:31.165643 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:31.167984 (+ 2341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:31.167991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:31.168022 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:31.168454 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.168469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.170610 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.170616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.173730 (+ 3114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.173747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.175877 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.175882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.176266 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.176349 (+ 83us) client_negotiation.cc:769] Sending connection context
0923 17:21:31.177467 (+ 1118us) client_negotiation.cc:240] Negotiation successful
0923 17:21:31.177630 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1205,"mutex_wait_us":131,"thread_start_us":1070,"threads_started":1}
I20260923 17:21:31.181967 31130 tablet_service.cc:1467] Processing CreateTablet for tablet 3608a23fd0c7469db40ab1fdb419412d (DEFAULT_TABLE table=test-table [id=9fd0c6e439ea442880d8d7279b687e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:31.181972 30990 tablet_service.cc:1467] Processing CreateTablet for tablet 3608a23fd0c7469db40ab1fdb419412d (DEFAULT_TABLE table=test-table [id=9fd0c6e439ea442880d8d7279b687e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:31.181972 30851 tablet_service.cc:1467] Processing CreateTablet for tablet 3608a23fd0c7469db40ab1fdb419412d (DEFAULT_TABLE table=test-table [id=9fd0c6e439ea442880d8d7279b687e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:31.183490 30851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3608a23fd0c7469db40ab1fdb419412d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:31.183523 30990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3608a23fd0c7469db40ab1fdb419412d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:31.183503 31130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3608a23fd0c7469db40ab1fdb419412d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:31.197028 31219 tablet_bootstrap.cc:492] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123: Bootstrap starting.
I20260923 17:21:31.200948 31219 tablet_bootstrap.cc:654] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:31.202101 31220 tablet_bootstrap.cc:492] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35: Bootstrap starting.
I20260923 17:21:31.202598 31219 log.cc:824] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:31.203279 31221 tablet_bootstrap.cc:492] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653: Bootstrap starting.
I20260923 17:21:31.206887 31219 tablet_bootstrap.cc:492] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123: No bootstrap required, opened a new log
I20260923 17:21:31.207269 31219 ts_tablet_manager.cc:1397] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260923 17:21:31.207588 31221 tablet_bootstrap.cc:654] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:31.208263 31220 tablet_bootstrap.cc:654] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:31.209076 31221 log.cc:824] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:31.209635 31220 log.cc:824] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:31.212633 31221 tablet_bootstrap.cc:492] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653: No bootstrap required, opened a new log
I20260923 17:21:31.212961 31221 ts_tablet_manager.cc:1397] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 17:21:31.213080 31220 tablet_bootstrap.cc:492] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35: No bootstrap required, opened a new log
I20260923 17:21:31.213363 31220 ts_tablet_manager.cc:1397] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260923 17:21:31.217700 31219 raft_consensus.cc:348] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } }
I20260923 17:21:31.218225 31219 raft_consensus.cc:374] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:31.218330 31219 raft_consensus.cc:729] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae72425ef074680b605c96678ced123, State: Initialized, Role: FOLLOWER
I20260923 17:21:31.219082 31219 consensus_queue.cc:260] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } }
I20260923 17:21:31.220429 31200 heartbeater.cc:503] Master 127.28.72.254:44581 was elected leader, sending a full tablet report...
I20260923 17:21:31.221143 31219 ts_tablet_manager.cc:1428] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260923 17:21:31.223716 31221 raft_consensus.cc:348] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } }
I20260923 17:21:31.223802 31220 raft_consensus.cc:348] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } }
I20260923 17:21:31.224176 31221 raft_consensus.cc:374] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:31.224263 31220 raft_consensus.cc:374] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:31.224298 31221 raft_consensus.cc:729] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b4fccd75ba448291d4ce8bc00a6653, State: Initialized, Role: FOLLOWER
I20260923 17:21:31.224359 31220 raft_consensus.cc:729] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50971779fba844e9b00cb0605df42e35, State: Initialized, Role: FOLLOWER
I20260923 17:21:31.225107 31220 consensus_queue.cc:260] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } }
I20260923 17:21:31.225149 31221 consensus_queue.cc:260] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } }
I20260923 17:21:31.228240 31220 ts_tablet_manager.cc:1428] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260923 17:21:31.228992 31221 ts_tablet_manager.cc:1428] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20260923 17:21:31.229279 31060 heartbeater.cc:503] Master 127.28.72.254:44581 was elected leader, sending a full tablet report...
W20260923 17:21:31.249293 31061 tablet.cc:2369] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:21:31.323681 31201 tablet.cc:2369] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:21:31.456874 30922 tablet.cc:2369] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:21:31.465965 31226 raft_consensus.cc:482] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:21:31.466341 31226 raft_consensus.cc:504] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } }
I20260923 17:21:31.468181 31226 leader_election.cc:288] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cae72425ef074680b605c96678ced123 (127.28.72.195:36001), 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194:34693)
I20260923 17:21:31.475225 31214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.468638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:47383 (local address 127.28.72.195:36001)
0923 17:21:31.468822 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:31.468827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:31.469602 (+ 775us) server_negotiation.cc:407] Connection header received
0923 17:21:31.469888 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:31.469895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:31.469981 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:31.470066 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:31.470762 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.471569 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.473391 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.474180 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.474411 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.474492 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:31.474776 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 17:21:31.474911 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:21:31.475407 31228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.468706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36001 (local address 127.28.72.193:47383)
0923 17:21:31.469426 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:31.469476 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:31.469663 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:31.470285 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:31.470290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:31.470318 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:31.470612 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.470620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.471703 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.471711 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.473214 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.473226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.474423 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.474429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.474606 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.474631 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:21:31.474748 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 17:21:31.474902 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":1,"thread_start_us":395,"threads_started":1}
I20260923 17:21:31.476282 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.469830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:41827 (local address 127.28.72.194:34693)
0923 17:21:31.470042 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:31.470047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:31.470206 (+ 159us) server_negotiation.cc:407] Connection header received
0923 17:21:31.470567 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:31.470571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:31.470636 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:31.470792 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:31.471500 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.472286 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.474208 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.475210 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.475433 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.475521 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:31.475908 (+ 387us) server_negotiation.cc:299] Negotiation successful
0923 17:21:31.476026 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:21:31.476372 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.469516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:34693 (local address 127.28.72.193:41827)
0923 17:21:31.470108 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:31.470127 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:31.470246 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:31.470808 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:31.470814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:31.470842 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:31.471317 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.471328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.472453 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.472460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.474023 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.474036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.475469 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.475476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.475679 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.475731 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 17:21:31.475857 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 17:21:31.476019 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":272,"threads_started":1}
I20260923 17:21:31.476794 31154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3608a23fd0c7469db40ab1fdb419412d" candidate_uuid: "50971779fba844e9b00cb0605df42e35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae72425ef074680b605c96678ced123" is_pre_election: true
I20260923 17:21:31.477358 31154 raft_consensus.cc:2454] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50971779fba844e9b00cb0605df42e35 in term 0.
I20260923 17:21:31.477591 31010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3608a23fd0c7469db40ab1fdb419412d" candidate_uuid: "50971779fba844e9b00cb0605df42e35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70b4fccd75ba448291d4ce8bc00a6653" is_pre_election: true
I20260923 17:21:31.478097 30802 leader_election.cc:302] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50971779fba844e9b00cb0605df42e35, cae72425ef074680b605c96678ced123; no voters:
I20260923 17:21:31.478251 31010 raft_consensus.cc:2454] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50971779fba844e9b00cb0605df42e35 in term 0.
I20260923 17:21:31.478586 31226 raft_consensus.cc:2781] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:21:31.478739 31226 raft_consensus.cc:482] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:21:31.478873 31226 raft_consensus.cc:3037] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:31.481359 31226 raft_consensus.cc:504] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } }
I20260923 17:21:31.482090 31226 leader_election.cc:288] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [CANDIDATE]: Term 1 election: Requested vote from peers cae72425ef074680b605c96678ced123 (127.28.72.195:36001), 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194:34693)
I20260923 17:21:31.482811 31154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3608a23fd0c7469db40ab1fdb419412d" candidate_uuid: "50971779fba844e9b00cb0605df42e35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae72425ef074680b605c96678ced123"
I20260923 17:21:31.482833 31010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3608a23fd0c7469db40ab1fdb419412d" candidate_uuid: "50971779fba844e9b00cb0605df42e35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70b4fccd75ba448291d4ce8bc00a6653"
I20260923 17:21:31.483029 31154 raft_consensus.cc:3037] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:31.483073 31010 raft_consensus.cc:3037] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:31.485387 31010 raft_consensus.cc:2454] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50971779fba844e9b00cb0605df42e35 in term 1.
I20260923 17:21:31.485383 31154 raft_consensus.cc:2454] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50971779fba844e9b00cb0605df42e35 in term 1.
I20260923 17:21:31.485946 30802 leader_election.cc:302] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50971779fba844e9b00cb0605df42e35, 70b4fccd75ba448291d4ce8bc00a6653; no voters:
I20260923 17:21:31.486335 31226 raft_consensus.cc:2781] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:31.486826 31226 raft_consensus.cc:686] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [term 1 LEADER]: Becoming Leader. State: Replica: 50971779fba844e9b00cb0605df42e35, State: Running, Role: LEADER
I20260923 17:21:31.487505 31226 consensus_queue.cc:237] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } }
I20260923 17:21:31.492009 30713 catalog_manager.cc:5495] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50971779fba844e9b00cb0605df42e35 (127.28.72.193). New cstate: current_term: 1 leader_uuid: "50971779fba844e9b00cb0605df42e35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } health_report { overall_health: HEALTHY } } }
Sep 23 17:21:31 dist-test-slave-j0m4 krb5kdc[30669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184089, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:31.579982 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.565791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33106 (local address 127.28.72.254:44581)
0923 17:21:31.566029 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:31.566036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:31.566585 (+ 549us) server_negotiation.cc:407] Connection header received
0923 17:21:31.566783 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:31.566787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:31.566846 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:31.566950 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:31.568135 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.568887 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.570292 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.570705 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.570847 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.574377 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:31.574382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:31.574385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:31.574417 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:31.577153 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:31.577914 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:31.577922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:31.577925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:31.578042 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:31.578401 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:31.578406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:31.578409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:31.578695 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:31.578849 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:31.579306 (+ 457us) server_negotiation.cc:299] Negotiation successful
0923 17:21:31.579574 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":117}
I20260923 17:21:31.580147 31237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.565822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:44581 (local address 127.0.0.1:33106)
0923 17:21:31.566468 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:31.566484 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:31.566661 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:31.566973 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:31.566977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:31.567741 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:31.567990 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.567999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.569026 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.569033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.570122 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.570135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.570892 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.570898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.571084 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.571856 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:31.571876 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:31.574130 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:31.577328 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:31.577342 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:31.577350 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:31.577794 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:31.578148 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:31.578155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:31.578158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:31.578283 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:31.578827 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:31.578836 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:31.579011 (+ 175us) client_negotiation.cc:769] Sending connection context
0923 17:21:31.579223 (+ 212us) client_negotiation.cc:240] Negotiation successful
0923 17:21:31.579699 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":182,"thread_start_us":319,"threads_started":1}
I20260923 17:21:31.585660 30713 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:33106:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:21:31.586877 30713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:21:31.593822 30851 tablet_service.cc:1467] Processing CreateTablet for tablet 64648781f3484cddba62d7c133caaaf7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=002e8ceb0bff48f58c3e79c7970d8dd8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:21:31.594163 31130 tablet_service.cc:1467] Processing CreateTablet for tablet 64648781f3484cddba62d7c133caaaf7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=002e8ceb0bff48f58c3e79c7970d8dd8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:21:31.594183 30990 tablet_service.cc:1467] Processing CreateTablet for tablet 64648781f3484cddba62d7c133caaaf7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=002e8ceb0bff48f58c3e79c7970d8dd8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:21:31.594928 30851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64648781f3484cddba62d7c133caaaf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:31.595042 31130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64648781f3484cddba62d7c133caaaf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:31.595042 30990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64648781f3484cddba62d7c133caaaf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:31.600890 31220 tablet_bootstrap.cc:492] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35: Bootstrap starting.
I20260923 17:21:31.602330 31219 tablet_bootstrap.cc:492] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123: Bootstrap starting.
I20260923 17:21:31.602363 31220 tablet_bootstrap.cc:654] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:31.603072 31221 tablet_bootstrap.cc:492] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653: Bootstrap starting.
I20260923 17:21:31.604522 31219 tablet_bootstrap.cc:654] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:31.604615 31221 tablet_bootstrap.cc:654] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:31.607383 31219 tablet_bootstrap.cc:492] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123: No bootstrap required, opened a new log
I20260923 17:21:31.607551 31219 ts_tablet_manager.cc:1397] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260923 17:21:31.607704 31221 tablet_bootstrap.cc:492] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653: No bootstrap required, opened a new log
I20260923 17:21:31.607848 31221 ts_tablet_manager.cc:1397] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:21:31.608246 31220 tablet_bootstrap.cc:492] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35: No bootstrap required, opened a new log
I20260923 17:21:31.608412 31220 ts_tablet_manager.cc:1397] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 17:21:31.608369 31219 raft_consensus.cc:348] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:31.608673 31219 raft_consensus.cc:374] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:31.608774 31219 raft_consensus.cc:729] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae72425ef074680b605c96678ced123, State: Initialized, Role: FOLLOWER
I20260923 17:21:31.608631 31221 raft_consensus.cc:348] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:31.608881 31221 raft_consensus.cc:374] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:31.608994 31221 raft_consensus.cc:729] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b4fccd75ba448291d4ce8bc00a6653, State: Initialized, Role: FOLLOWER
I20260923 17:21:31.609048 31219 consensus_queue.cc:260] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:31.609200 31220 raft_consensus.cc:348] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:31.609452 31220 raft_consensus.cc:374] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:31.609541 31220 raft_consensus.cc:729] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50971779fba844e9b00cb0605df42e35, State: Initialized, Role: FOLLOWER
I20260923 17:21:31.609674 31219 ts_tablet_manager.cc:1428] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260923 17:21:31.609555 31225 tablet_replica.cc:439] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } }
I20260923 17:21:31.609740 31220 consensus_queue.cc:260] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:31.609917 31225 tablet_replica.cc:442] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:21:31.609234 31221 consensus_queue.cc:260] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:31.610351 31220 ts_tablet_manager.cc:1428] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:21:31.610304 31226 tablet_replica.cc:439] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } }
I20260923 17:21:31.610596 31226 tablet_replica.cc:442] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:21:31.610600 31227 tablet_replica.cc:439] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } }
I20260923 17:21:31.610952 31227 tablet_replica.cc:442] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:21:31.611483 31221 ts_tablet_manager.cc:1428] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260923 17:21:31.747609 31226 raft_consensus.cc:482] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:21:31.747829 31226 raft_consensus.cc:504] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:31.748620 31226 leader_election.cc:288] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194:34693), cae72425ef074680b605c96678ced123 (127.28.72.195:36001)
I20260923 17:21:31.749320 31010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64648781f3484cddba62d7c133caaaf7" candidate_uuid: "50971779fba844e9b00cb0605df42e35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70b4fccd75ba448291d4ce8bc00a6653" is_pre_election: true
I20260923 17:21:31.749408 31154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64648781f3484cddba62d7c133caaaf7" candidate_uuid: "50971779fba844e9b00cb0605df42e35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae72425ef074680b605c96678ced123" is_pre_election: true
I20260923 17:21:31.749639 31010 raft_consensus.cc:2454] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50971779fba844e9b00cb0605df42e35 in term 0.
I20260923 17:21:31.749657 31154 raft_consensus.cc:2454] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50971779fba844e9b00cb0605df42e35 in term 0.
I20260923 17:21:31.750077 30802 leader_election.cc:302] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50971779fba844e9b00cb0605df42e35, 70b4fccd75ba448291d4ce8bc00a6653; no voters:
I20260923 17:21:31.750360 31226 raft_consensus.cc:2781] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:21:31.750483 31226 raft_consensus.cc:482] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:21:31.750691 31226 raft_consensus.cc:3037] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:31.752821 31226 raft_consensus.cc:504] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:31.753372 31226 leader_election.cc:288] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [CANDIDATE]: Term 1 election: Requested vote from peers 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194:34693), cae72425ef074680b605c96678ced123 (127.28.72.195:36001)
I20260923 17:21:31.753926 31010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64648781f3484cddba62d7c133caaaf7" candidate_uuid: "50971779fba844e9b00cb0605df42e35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70b4fccd75ba448291d4ce8bc00a6653"
I20260923 17:21:31.753970 31154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64648781f3484cddba62d7c133caaaf7" candidate_uuid: "50971779fba844e9b00cb0605df42e35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae72425ef074680b605c96678ced123"
I20260923 17:21:31.754169 31010 raft_consensus.cc:3037] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:31.754191 31154 raft_consensus.cc:3037] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:31.757160 31010 raft_consensus.cc:2454] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50971779fba844e9b00cb0605df42e35 in term 1.
I20260923 17:21:31.757161 31154 raft_consensus.cc:2454] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50971779fba844e9b00cb0605df42e35 in term 1.
I20260923 17:21:31.757618 30802 leader_election.cc:302] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50971779fba844e9b00cb0605df42e35, 70b4fccd75ba448291d4ce8bc00a6653; no voters:
I20260923 17:21:31.757881 31226 raft_consensus.cc:2781] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:31.758068 31226 raft_consensus.cc:686] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [term 1 LEADER]: Becoming Leader. State: Replica: 50971779fba844e9b00cb0605df42e35, State: Running, Role: LEADER
I20260923 17:21:31.758375 31226 consensus_queue.cc:237] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:31.760073 31230 tablet_replica.cc:439] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35: TxnStatusTablet state changed. Reason: New leader 50971779fba844e9b00cb0605df42e35. Latest consensus state: current_term: 1 leader_uuid: "50971779fba844e9b00cb0605df42e35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } }
I20260923 17:21:31.760434 31230 tablet_replica.cc:442] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:21:31.761070 31242 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:21:31.761341 31242 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:21:31.761543 30713 catalog_manager.cc:5495] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50971779fba844e9b00cb0605df42e35 (127.28.72.193). New cstate: current_term: 1 leader_uuid: "50971779fba844e9b00cb0605df42e35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } health_report { overall_health: UNKNOWN } } }
I20260923 17:21:31.804765 31237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.797755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:41835 (local address 127.0.0.1:58246)
0923 17:21:31.798015 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:31.798040 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:31.798230 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:31.799514 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:31.799519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:31.799536 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:21:31.799766 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.799777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.800934 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.800940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.801934 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.801947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.802742 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.802750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.802968 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.803031 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:21:31.804177 (+ 1146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:21:31.804200 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:21:31.804287 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 17:21:31.804474 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 17:21:31.804811 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.797949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.28.72.193:41835)
0923 17:21:31.799027 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:31.799031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:31.799051 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:21:31.799211 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:31.799215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:31.799298 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:31.799490 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:31.799957 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.800784 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.802091 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.802552 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.802701 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.803325 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:31.804063 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:31.804222 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:31.804303 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 17:21:31.804386 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":787,"threads_started":1}
I20260923 17:21:31.813228 31010 raft_consensus.cc:1261] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Refusing update from remote peer 50971779fba844e9b00cb0605df42e35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:31.813333 31154 raft_consensus.cc:1261] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Refusing update from remote peer 50971779fba844e9b00cb0605df42e35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:31.814385 31226 consensus_queue.cc:1035] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:31.814829 31230 consensus_queue.cc:1035] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:31.826606 31225 tablet_replica.cc:439] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123: TxnStatusTablet state changed. Reason: New leader 50971779fba844e9b00cb0605df42e35. Latest consensus state: current_term: 1 leader_uuid: "50971779fba844e9b00cb0605df42e35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } }
I20260923 17:21:31.826915 31225 tablet_replica.cc:442] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:21:31.829741 31246 mvcc.cc:205] Tried to move back new op lower bound from 7332594040055750656 to 7332594039844884480. Current Snapshot: MvccSnapshot[applied={T|T < 7332594040055750656}]
I20260923 17:21:31.830592 31230 tablet_replica.cc:439] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50971779fba844e9b00cb0605df42e35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } }
I20260923 17:21:31.830868 31230 tablet_replica.cc:442] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:21:31.831426 31247 mvcc.cc:205] Tried to move back new op lower bound from 7332594040055750656 to 7332594039844884480. Current Snapshot: MvccSnapshot[applied={T|T < 7332594040055750656}]
I20260923 17:21:31.836243 31227 tablet_replica.cc:439] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653: TxnStatusTablet state changed. Reason: New leader 50971779fba844e9b00cb0605df42e35. Latest consensus state: current_term: 1 leader_uuid: "50971779fba844e9b00cb0605df42e35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } }
I20260923 17:21:31.836570 31227 tablet_replica.cc:442] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:21:31.838781 31226 tablet_replica.cc:439] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50971779fba844e9b00cb0605df42e35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } }
I20260923 17:21:31.839212 31226 tablet_replica.cc:442] T 64648781f3484cddba62d7c133caaaf7 P 50971779fba844e9b00cb0605df42e35: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:21:31.843384 31225 tablet_replica.cc:439] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50971779fba844e9b00cb0605df42e35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } }
I20260923 17:21:31.843647 31225 tablet_replica.cc:442] T 64648781f3484cddba62d7c133caaaf7 P cae72425ef074680b605c96678ced123: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:21:31.854137 31227 tablet_replica.cc:439] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50971779fba844e9b00cb0605df42e35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } } }
I20260923 17:21:31.854439 31227 tablet_replica.cc:442] T 64648781f3484cddba62d7c133caaaf7 P 70b4fccd75ba448291d4ce8bc00a6653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:21:31.861392 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.842894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.28.72.193:41835)
0923 17:21:31.843071 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:31.843080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:31.852638 (+ 9558us) server_negotiation.cc:407] Connection header received
0923 17:21:31.855502 (+ 2864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:31.855508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:31.855607 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:31.855705 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:31.856506 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.857683 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.859271 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.859944 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.860075 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.860568 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:31.860706 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:31.860799 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:31.860984 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 17:21:31.861068 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:21:31.881564 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.874051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:57879 (local address 127.28.72.193:41835)
0923 17:21:31.874294 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:31.874302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:31.874962 (+ 660us) server_negotiation.cc:407] Connection header received
0923 17:21:31.875231 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:31.875234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:31.875310 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:31.875395 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:31.876042 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.876780 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.879088 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.879797 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.879922 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.880643 (+ 721us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:31.880779 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:31.880858 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:31.881038 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 17:21:31.881171 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
I20260923 17:21:31.881852 31259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:31.874060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:41835 (local address 127.28.72.193:57879)
0923 17:21:31.874818 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:31.874844 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:31.875002 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:31.875434 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:31.875439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:31.875452 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:21:31.875796 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.875807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.876922 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.876932 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.878886 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:31.878910 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:31.880008 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:31.880015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:31.880213 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:31.880314 (+ 101us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:21:31.880881 (+ 567us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:21:31.880893 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 17:21:31.880982 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 17:21:31.881143 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":393,"threads_started":1}
I20260923 17:21:31.892907 31154 raft_consensus.cc:1261] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Refusing update from remote peer 50971779fba844e9b00cb0605df42e35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:31.893537 31010 raft_consensus.cc:1261] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Refusing update from remote peer 50971779fba844e9b00cb0605df42e35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:31.893837 31226 consensus_queue.cc:1035] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:31.894966 31226 consensus_queue.cc:1035] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:31.979166 30712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:31.979487 30712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:31.983676 30712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c: Sending DeleteTablet for 3 replicas of tablet 3608a23fd0c7469db40ab1fdb419412d
I20260923 17:21:31.984987 31130 tablet_service.cc:1514] Processing DeleteTablet for tablet 3608a23fd0c7469db40ab1fdb419412d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:31 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:46006
I20260923 17:21:31.984989 30851 tablet_service.cc:1514] Processing DeleteTablet for tablet 3608a23fd0c7469db40ab1fdb419412d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:31 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:58240
I20260923 17:21:31.985946 31263 tablet_replica.cc:330] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35: stopping tablet replica
I20260923 17:21:31.986020 30990 tablet_service.cc:1514] Processing DeleteTablet for tablet 3608a23fd0c7469db40ab1fdb419412d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:31 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:50538
I20260923 17:21:31.986639 31263 raft_consensus.cc:2229] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:21:31.987509 31263 raft_consensus.cc:2258] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:31.987468 30712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:21:31.988293 30712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:21:31.990564 31263 ts_tablet_manager.cc:1905] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:31.996701 31129 tablet_service.cc:1467] Processing CreateTablet for tablet bab697469b664e27897de27b5debebee (DEFAULT_TABLE table=test-table [id=95338f130e7f41168302447fabc25d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:31.998373 30989 tablet_service.cc:1467] Processing CreateTablet for tablet bab697469b664e27897de27b5debebee (DEFAULT_TABLE table=test-table [id=95338f130e7f41168302447fabc25d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:31.998912 30989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab697469b664e27897de27b5debebee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:31.999217 31129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab697469b664e27897de27b5debebee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:32.001405 31263 ts_tablet_manager.cc:1918] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:21:32.001720 31263 log.cc:1196] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/3608a23fd0c7469db40ab1fdb419412d
I20260923 17:21:32.002517 31263 ts_tablet_manager.cc:1939] T 3608a23fd0c7469db40ab1fdb419412d P 50971779fba844e9b00cb0605df42e35: Deleting consensus metadata
I20260923 17:21:32.003695 30850 tablet_service.cc:1467] Processing CreateTablet for tablet bab697469b664e27897de27b5debebee (DEFAULT_TABLE table=test-table [id=95338f130e7f41168302447fabc25d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:32.004264 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab697469b664e27897de27b5debebee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:32.007491 30700 catalog_manager.cc:4841] TS 50971779fba844e9b00cb0605df42e35 (127.28.72.193:41835): tablet 3608a23fd0c7469db40ab1fdb419412d (table test-table [id=9fd0c6e439ea442880d8d7279b687e64]) successfully deleted
I20260923 17:21:32.011461 31220 tablet_bootstrap.cc:492] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35: Bootstrap starting.
I20260923 17:21:32.013334 31220 tablet_bootstrap.cc:654] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:32.013896 31265 tablet_replica.cc:330] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653: stopping tablet replica
I20260923 17:21:32.014400 31265 raft_consensus.cc:2229] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:32.014787 31265 raft_consensus.cc:2258] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:32.015692 31264 tablet_replica.cc:330] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123: stopping tablet replica
I20260923 17:21:32.017313 31264 raft_consensus.cc:2229] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:32.017714 31264 raft_consensus.cc:2258] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:32.018705 31219 tablet_bootstrap.cc:492] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123: Bootstrap starting.
I20260923 17:21:32.018891 31265 ts_tablet_manager.cc:1905] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:32.020532 31221 tablet_bootstrap.cc:492] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653: Bootstrap starting.
I20260923 17:21:32.020793 31219 tablet_bootstrap.cc:654] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:32.022133 31221 tablet_bootstrap.cc:654] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:32.024843 31221 tablet_bootstrap.cc:492] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653: No bootstrap required, opened a new log
I20260923 17:21:32.025025 31221 ts_tablet_manager.cc:1397] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 17:21:32.025731 31221 raft_consensus.cc:348] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:32.025954 31221 raft_consensus.cc:374] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:32.026023 31221 raft_consensus.cc:729] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b4fccd75ba448291d4ce8bc00a6653, State: Initialized, Role: FOLLOWER
I20260923 17:21:32.026283 31221 consensus_queue.cc:260] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:32.026922 31221 ts_tablet_manager.cc:1428] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:21:32.028199 31220 tablet_bootstrap.cc:492] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35: No bootstrap required, opened a new log
I20260923 17:21:32.028393 31220 ts_tablet_manager.cc:1397] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.005s
I20260923 17:21:32.029183 31220 raft_consensus.cc:348] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:32.029436 31220 raft_consensus.cc:374] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:32.029515 31220 raft_consensus.cc:729] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50971779fba844e9b00cb0605df42e35, State: Initialized, Role: FOLLOWER
I20260923 17:21:32.029927 31264 ts_tablet_manager.cc:1905] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:32.029796 31220 consensus_queue.cc:260] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:32.030563 31220 ts_tablet_manager.cc:1428] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:21:32.036972 31219 tablet_bootstrap.cc:492] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123: No bootstrap required, opened a new log
I20260923 17:21:32.037174 31219 ts_tablet_manager.cc:1397] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.004s
I20260923 17:21:32.037659 31264 ts_tablet_manager.cc:1918] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:21:32.037842 31264 log.cc:1196] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/3608a23fd0c7469db40ab1fdb419412d
I20260923 17:21:32.037976 31219 raft_consensus.cc:348] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:32.038244 31219 raft_consensus.cc:374] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:32.038328 31219 raft_consensus.cc:729] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae72425ef074680b605c96678ced123, State: Initialized, Role: FOLLOWER
I20260923 17:21:32.038499 31264 ts_tablet_manager.cc:1939] T 3608a23fd0c7469db40ab1fdb419412d P cae72425ef074680b605c96678ced123: Deleting consensus metadata
I20260923 17:21:32.038672 31219 consensus_queue.cc:260] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:32.041921 31219 ts_tablet_manager.cc:1428] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260923 17:21:32.044849 31265 ts_tablet_manager.cc:1918] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:21:32.045053 31265 log.cc:1196] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/3608a23fd0c7469db40ab1fdb419412d
I20260923 17:21:32.045492 31265 ts_tablet_manager.cc:1939] T 3608a23fd0c7469db40ab1fdb419412d P 70b4fccd75ba448291d4ce8bc00a6653: Deleting consensus metadata
I20260923 17:21:32.047173 30699 catalog_manager.cc:4841] TS 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194:34693): tablet 3608a23fd0c7469db40ab1fdb419412d (table test-table [id=9fd0c6e439ea442880d8d7279b687e64]) successfully deleted
I20260923 17:21:32.048190 30699 catalog_manager.cc:4841] TS cae72425ef074680b605c96678ced123 (127.28.72.195:36001): tablet 3608a23fd0c7469db40ab1fdb419412d (table test-table [id=9fd0c6e439ea442880d8d7279b687e64]) successfully deleted
I20260923 17:21:32.193670 31225 raft_consensus.cc:482] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:21:32.193987 31225 raft_consensus.cc:504] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:32.196091 31225 leader_election.cc:288] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50971779fba844e9b00cb0605df42e35 (127.28.72.193:41835), 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194:34693)
I20260923 17:21:32.209079 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:32.196781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:35419 (local address 127.28.72.193:41835)
0923 17:21:32.197101 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:32.197111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:32.197727 (+ 616us) server_negotiation.cc:407] Connection header received
0923 17:21:32.198115 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:32.198124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:32.198244 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:32.198337 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:32.199505 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.201111 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.205198 (+ 4087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.207246 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.207475 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:32.207598 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:32.208577 (+ 979us) server_negotiation.cc:299] Negotiation successful
0923 17:21:32.208771 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 17:21:32.209561 31269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:32.196537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:41835 (local address 127.28.72.195:35419)
0923 17:21:32.197575 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:32.197603 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:32.197872 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:32.198639 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:32.198648 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:32.198686 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:32.199208 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:32.199226 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.201472 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.201481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:32.204903 (+ 3422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:32.204936 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.207976 (+ 3040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.207987 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:32.208283 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:32.208340 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 17:21:32.209053 (+ 713us) client_negotiation.cc:240] Negotiation successful
0923 17:21:32.209218 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"thread_start_us":677,"threads_started":1}
I20260923 17:21:32.210520 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab697469b664e27897de27b5debebee" candidate_uuid: "cae72425ef074680b605c96678ced123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50971779fba844e9b00cb0605df42e35" is_pre_election: true
I20260923 17:21:32.211025 30871 raft_consensus.cc:2454] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae72425ef074680b605c96678ced123 in term 0.
I20260923 17:21:32.211789 31082 leader_election.cc:302] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50971779fba844e9b00cb0605df42e35, cae72425ef074680b605c96678ced123; no voters:
I20260923 17:21:32.212298 31225 raft_consensus.cc:2781] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:21:32.212442 31225 raft_consensus.cc:482] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:21:32.212567 31225 raft_consensus.cc:3037] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:32.216040 31225 raft_consensus.cc:504] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:32.216810 31225 leader_election.cc:288] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [CANDIDATE]: Term 1 election: Requested vote from peers 50971779fba844e9b00cb0605df42e35 (127.28.72.193:41835), 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194:34693)
I20260923 17:21:32.217442 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab697469b664e27897de27b5debebee" candidate_uuid: "cae72425ef074680b605c96678ced123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50971779fba844e9b00cb0605df42e35"
I20260923 17:21:32.217762 30871 raft_consensus.cc:3037] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:32.219209 31271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:32.196541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:34693 (local address 127.28.72.195:43395)
0923 17:21:32.209924 (+ 13383us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:32.209952 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:32.210124 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:32.211197 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:32.211203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:32.211234 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:32.211680 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:32.211693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.213226 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.213233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:32.215555 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:32.215574 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.218236 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.218243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:32.218488 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:32.218524 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:21:32.218632 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 17:21:32.218785 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12369,"mutex_wait_us":203,"thread_start_us":12003,"threads_started":1}
I20260923 17:21:32.219316 31270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:32.196630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:43395 (local address 127.28.72.194:34693)
0923 17:21:32.210685 (+ 14055us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:32.210690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:32.210710 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:21:32.210766 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:32.210918 (+ 152us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:32.211013 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:32.211167 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:32.211907 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.213065 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.215756 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.217737 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.217948 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:32.218072 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:32.218708 (+ 636us) server_negotiation.cc:299] Negotiation successful
0923 17:21:32.218920 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13916,"thread_start_us":13757,"threads_started":1}
I20260923 17:21:32.220110 31010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab697469b664e27897de27b5debebee" candidate_uuid: "cae72425ef074680b605c96678ced123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70b4fccd75ba448291d4ce8bc00a6653" is_pre_election: true
I20260923 17:21:32.220362 31010 raft_consensus.cc:2454] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae72425ef074680b605c96678ced123 in term 0.
I20260923 17:21:32.220759 31009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab697469b664e27897de27b5debebee" candidate_uuid: "cae72425ef074680b605c96678ced123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70b4fccd75ba448291d4ce8bc00a6653"
I20260923 17:21:32.220975 31009 raft_consensus.cc:3037] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:32.224274 31009 raft_consensus.cc:2454] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae72425ef074680b605c96678ced123 in term 1.
I20260923 17:21:32.224949 31081 leader_election.cc:302] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70b4fccd75ba448291d4ce8bc00a6653, cae72425ef074680b605c96678ced123; no voters:
I20260923 17:21:32.225355 31225 raft_consensus.cc:2781] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:32.225528 30871 raft_consensus.cc:2454] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae72425ef074680b605c96678ced123 in term 1.
I20260923 17:21:32.226140 31225 raft_consensus.cc:686] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [term 1 LEADER]: Becoming Leader. State: Replica: cae72425ef074680b605c96678ced123, State: Running, Role: LEADER
I20260923 17:21:32.226629 31225 consensus_queue.cc:237] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:32.230633 30712 catalog_manager.cc:5495] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 reported cstate change: term changed from 0 to 1, leader changed from <none> to cae72425ef074680b605c96678ced123 (127.28.72.195). New cstate: current_term: 1 leader_uuid: "cae72425ef074680b605c96678ced123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } health_report { overall_health: HEALTHY } } }
I20260923 17:21:32.288479 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:32.282069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46014 (local address 127.28.72.195:36001)
0923 17:21:32.282735 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:32.282740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:32.282758 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:21:32.282859 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:32.282862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:32.282941 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:32.283126 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:32.283626 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.284426 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.285855 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.286268 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.286419 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:32.286882 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:32.287667 (+ 785us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:32.287829 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:32.287932 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 17:21:32.288018 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":385,"threads_started":1}
I20260923 17:21:32.309026 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:32.301805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:46021 (local address 127.28.72.193:41835)
0923 17:21:32.302049 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:32.302059 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:32.302679 (+ 620us) server_negotiation.cc:407] Connection header received
0923 17:21:32.302915 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:32.302919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:32.302985 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:32.303076 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:32.303756 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.304588 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.306924 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.307577 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.307680 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:32.308281 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:32.308465 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:32.308536 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:32.308703 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 17:21:32.308776 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 17:21:32.309371 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:32.301763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:41835 (local address 127.28.72.195:46021)
0923 17:21:32.302552 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:32.302575 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:32.302757 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:32.303145 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:32.303151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:32.303165 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:21:32.303552 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:32.303566 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.304738 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.304744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:32.306733 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:32.306753 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.307783 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.307791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:32.308010 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:32.308099 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:21:32.308572 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:21:32.308586 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 17:21:32.308682 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 17:21:32.308924 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":370,"threads_started":1}
I20260923 17:21:32.319393 31009 raft_consensus.cc:1261] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Refusing update from remote peer cae72425ef074680b605c96678ced123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:32.319617 30871 raft_consensus.cc:1261] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Refusing update from remote peer cae72425ef074680b605c96678ced123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:32.320235 31272 consensus_queue.cc:1035] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:32.320708 31225 consensus_queue.cc:1035] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:32.335742 31248 mvcc.cc:205] Tried to move back new op lower bound from 7332594042132819968 to 7332594041765011456. Current Snapshot: MvccSnapshot[applied={T|T < 7332594042132819968}]
I20260923 17:21:32.359683 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:32.352850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:56929 (local address 127.28.72.195:36001)
0923 17:21:32.353030 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:32.353036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:32.353179 (+ 143us) server_negotiation.cc:407] Connection header received
0923 17:21:32.353384 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:32.353389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:32.353473 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:32.353578 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:32.353992 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.355254 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.357466 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.358072 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.358241 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:32.358795 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:32.358947 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:32.359043 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:32.359277 (+ 234us) server_negotiation.cc:299] Negotiation successful
0923 17:21:32.359373 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:21:32.359767 31259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:32.352855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36001 (local address 127.28.72.193:56929)
0923 17:21:32.353075 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:32.353097 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:32.353235 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:32.353596 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:32.353600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:32.353612 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:21:32.353829 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:32.353839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.355449 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.355455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:32.357237 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:32.357265 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:32.358263 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:32.358270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:32.358552 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:32.358599 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:21:32.359078 (+ 479us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:21:32.359095 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 17:21:32.359235 (+ 140us) client_negotiation.cc:240] Negotiation successful
0923 17:21:32.359421 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 17:21:32.360984 31247 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 17:21:33.968600 30712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:33.968881 30712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:33.973838 30712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c: Sending DeleteTablet for 3 replicas of tablet bab697469b664e27897de27b5debebee
I20260923 17:21:33.974788 30850 tablet_service.cc:1514] Processing DeleteTablet for tablet bab697469b664e27897de27b5debebee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:33 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:58240
I20260923 17:21:33.974996 30989 tablet_service.cc:1514] Processing DeleteTablet for tablet bab697469b664e27897de27b5debebee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:33 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:50538
I20260923 17:21:33.975282 31130 tablet_service.cc:1514] Processing DeleteTablet for tablet bab697469b664e27897de27b5debebee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:33 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:46006
I20260923 17:21:33.987913 31304 tablet_replica.cc:330] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35: stopping tablet replica
I20260923 17:21:33.988521 31304 raft_consensus.cc:2229] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:33.989027 31304 raft_consensus.cc:2258] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:33.990355 31305 tablet_replica.cc:330] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653: stopping tablet replica
I20260923 17:21:33.990538 31304 ts_tablet_manager.cc:1905] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:33.990607 31305 raft_consensus.cc:2229] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:33.990869 31305 raft_consensus.cc:2258] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:33.992363 31305 ts_tablet_manager.cc:1905] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:33.999482 31304 ts_tablet_manager.cc:1918] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:21:33.999550 31305 ts_tablet_manager.cc:1918] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:21:33.999670 31305 log.cc:1196] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/bab697469b664e27897de27b5debebee
I20260923 17:21:33.999646 31306 tablet_replica.cc:330] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123: stopping tablet replica
I20260923 17:21:33.999671 31304 log.cc:1196] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/bab697469b664e27897de27b5debebee
I20260923 17:21:33.999857 31306 raft_consensus.cc:2229] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:21:34.000136 31305 ts_tablet_manager.cc:1939] T bab697469b664e27897de27b5debebee P 70b4fccd75ba448291d4ce8bc00a6653: Deleting consensus metadata
I20260923 17:21:34.000175 31304 ts_tablet_manager.cc:1939] T bab697469b664e27897de27b5debebee P 50971779fba844e9b00cb0605df42e35: Deleting consensus metadata
I20260923 17:21:34.000471 31306 raft_consensus.cc:2258] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:34.001520 31306 ts_tablet_manager.cc:1905] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:34.001531 30699 catalog_manager.cc:4841] TS 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194:34693): tablet bab697469b664e27897de27b5debebee (table test-table [id=95338f130e7f41168302447fabc25d9a]) successfully deleted
I20260923 17:21:34.001531 30700 catalog_manager.cc:4841] TS 50971779fba844e9b00cb0605df42e35 (127.28.72.193:41835): tablet bab697469b664e27897de27b5debebee (table test-table [id=95338f130e7f41168302447fabc25d9a]) successfully deleted
I20260923 17:21:34.006915 31306 ts_tablet_manager.cc:1918] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:21:34.007153 31306 log.cc:1196] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/bab697469b664e27897de27b5debebee
I20260923 17:21:34.007577 31306 ts_tablet_manager.cc:1939] T bab697469b664e27897de27b5debebee P cae72425ef074680b605c96678ced123: Deleting consensus metadata
I20260923 17:21:34.008872 30699 catalog_manager.cc:4841] TS cae72425ef074680b605c96678ced123 (127.28.72.195:36001): tablet bab697469b664e27897de27b5debebee (table test-table [id=95338f130e7f41168302447fabc25d9a]) successfully deleted
Sep 23 17:21:34 dist-test-slave-j0m4 krb5kdc[30669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184094, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:21:34 dist-test-slave-j0m4 krb5kdc[30669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184094, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:34.045480 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:34.029324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.28.72.254:44581)
0923 17:21:34.029918 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:34.029922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:34.029945 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:21:34.030105 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:34.030109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:34.030172 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:34.030268 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:34.031813 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.032772 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.034124 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.034604 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.034716 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:34.039674 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:34.039680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:34.039683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:34.039724 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:34.042486 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:34.043439 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:34.043447 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:34.043451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:34.043596 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:34.043999 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:34.044003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:34.044005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:34.044326 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:34.044571 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:34.044891 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 17:21:34.045124 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":304,"threads_started":1}
I20260923 17:21:34.049259 30712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:21:34.049961 30712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:21:34.056993 30850 tablet_service.cc:1467] Processing CreateTablet for tablet 79ea811d9af348cda1528b3deb3e8b2d (DEFAULT_TABLE table=test-table [id=434cb1e661d242bdb8cbb2a87403bc68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:34.056994 30989 tablet_service.cc:1467] Processing CreateTablet for tablet 79ea811d9af348cda1528b3deb3e8b2d (DEFAULT_TABLE table=test-table [id=434cb1e661d242bdb8cbb2a87403bc68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:34.057238 31130 tablet_service.cc:1467] Processing CreateTablet for tablet 79ea811d9af348cda1528b3deb3e8b2d (DEFAULT_TABLE table=test-table [id=434cb1e661d242bdb8cbb2a87403bc68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:34.057471 30989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ea811d9af348cda1528b3deb3e8b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:34.057471 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ea811d9af348cda1528b3deb3e8b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:34.057725 31130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ea811d9af348cda1528b3deb3e8b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:34.065310 31317 tablet_bootstrap.cc:492] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35: Bootstrap starting.
I20260923 17:21:34.065775 31315 tablet_bootstrap.cc:492] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123: Bootstrap starting.
I20260923 17:21:34.067157 31317 tablet_bootstrap.cc:654] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:34.068171 31315 tablet_bootstrap.cc:654] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:34.068753 31316 tablet_bootstrap.cc:492] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653: Bootstrap starting.
I20260923 17:21:34.070933 31316 tablet_bootstrap.cc:654] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:34.070405 31317 tablet_bootstrap.cc:492] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35: No bootstrap required, opened a new log
I20260923 17:21:34.071790 31317 ts_tablet_manager.cc:1397] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260923 17:21:34.072903 31317 raft_consensus.cc:348] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.073247 31317 raft_consensus.cc:374] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:34.073360 31317 raft_consensus.cc:729] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50971779fba844e9b00cb0605df42e35, State: Initialized, Role: FOLLOWER
I20260923 17:21:34.073714 31317 consensus_queue.cc:260] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.078028 31317 ts_tablet_manager.cc:1428] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260923 17:21:34.083174 31315 tablet_bootstrap.cc:492] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123: No bootstrap required, opened a new log
I20260923 17:21:34.083345 31315 ts_tablet_manager.cc:1397] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.006s
I20260923 17:21:34.084170 31315 raft_consensus.cc:348] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.084450 31315 raft_consensus.cc:374] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:34.084554 31315 raft_consensus.cc:729] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae72425ef074680b605c96678ced123, State: Initialized, Role: FOLLOWER
I20260923 17:21:34.084843 31315 consensus_queue.cc:260] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.085402 31315 ts_tablet_manager.cc:1428] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 17:21:34.087219 31316 tablet_bootstrap.cc:492] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653: No bootstrap required, opened a new log
I20260923 17:21:34.087468 31316 ts_tablet_manager.cc:1397] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20260923 17:21:34.088389 31316 raft_consensus.cc:348] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.088690 31316 raft_consensus.cc:374] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:34.088796 31316 raft_consensus.cc:729] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b4fccd75ba448291d4ce8bc00a6653, State: Initialized, Role: FOLLOWER
I20260923 17:21:34.089042 31316 consensus_queue.cc:260] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.089861 31316 ts_tablet_manager.cc:1428] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:21:34.412417 31321 raft_consensus.cc:482] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:21:34.412701 31321 raft_consensus.cc:504] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.414511 31321 leader_election.cc:288] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50971779fba844e9b00cb0605df42e35 (127.28.72.193:41835), cae72425ef074680b605c96678ced123 (127.28.72.195:36001)
I20260923 17:21:34.434624 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:34.415040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:41941 (local address 127.28.72.195:36001)
0923 17:21:34.427549 (+ 12509us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:34.427559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:34.428194 (+ 635us) server_negotiation.cc:407] Connection header received
0923 17:21:34.428379 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:34.428383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:34.428444 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:34.428562 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:34.429223 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.430075 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.432432 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.433503 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.433662 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:34.433749 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:34.434107 (+ 358us) server_negotiation.cc:299] Negotiation successful
0923 17:21:34.434257 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12288,"thread_start_us":12140,"threads_started":1}
I20260923 17:21:34.434643 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:34.414915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:35021 (local address 127.28.72.193:41835)
0923 17:21:34.428460 (+ 13545us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:34.428464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:34.428485 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:21:34.428542 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:34.428545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:34.428601 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:34.428733 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:34.429442 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.430224 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.432263 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.433554 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.433775 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:34.433856 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:34.434159 (+ 303us) server_negotiation.cc:299] Negotiation successful
0923 17:21:34.434318 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13425,"thread_start_us":13279,"threads_started":1}
I20260923 17:21:34.434719 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:34.414981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36001 (local address 127.28.72.194:41941)
0923 17:21:34.428067 (+ 13086us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:34.428088 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:34.428210 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:34.428585 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:34.428590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:34.428618 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:34.429009 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:34.429024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.430232 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.430238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:34.432245 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:34.432262 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.433693 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.433700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:34.433946 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:34.433975 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 17:21:34.434051 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 17:21:34.434239 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12951,"mutex_wait_us":34,"spinlock_wait_cycles":8960,"thread_start_us":12386,"threads_started":1}
I20260923 17:21:34.434775 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:34.414721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:41835 (local address 127.28.72.194:35021)
0923 17:21:34.427751 (+ 13030us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:34.427784 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:34.428021 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:34.428767 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:34.428771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:34.428835 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:34.429270 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:34.429279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.430360 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.430366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:34.432058 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:34.432073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.433792 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.433800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:34.433971 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:34.433996 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:21:34.434107 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 17:21:34.434305 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12843,"mutex_wait_us":50,"spinlock_wait_cycles":1664,"thread_start_us":13092,"threads_started":1}
I20260923 17:21:34.435632 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ea811d9af348cda1528b3deb3e8b2d" candidate_uuid: "70b4fccd75ba448291d4ce8bc00a6653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50971779fba844e9b00cb0605df42e35" is_pre_election: true
I20260923 17:21:34.435993 30871 raft_consensus.cc:2454] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70b4fccd75ba448291d4ce8bc00a6653 in term 0.
I20260923 17:21:34.435916 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ea811d9af348cda1528b3deb3e8b2d" candidate_uuid: "70b4fccd75ba448291d4ce8bc00a6653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae72425ef074680b605c96678ced123" is_pre_election: true
I20260923 17:21:34.436172 31295 raft_consensus.cc:482] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:21:34.436316 31295 raft_consensus.cc:504] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.436540 30943 leader_election.cc:302] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50971779fba844e9b00cb0605df42e35, 70b4fccd75ba448291d4ce8bc00a6653; no voters:
I20260923 17:21:34.436735 31153 raft_consensus.cc:2454] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70b4fccd75ba448291d4ce8bc00a6653 in term 0.
I20260923 17:21:34.436904 31321 raft_consensus.cc:2781] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:21:34.437052 31321 raft_consensus.cc:482] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:21:34.437150 31321 raft_consensus.cc:3037] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:34.437127 31295 leader_election.cc:288] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50971779fba844e9b00cb0605df42e35 (127.28.72.193:41835), 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194:34693)
I20260923 17:21:34.437599 31010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ea811d9af348cda1528b3deb3e8b2d" candidate_uuid: "cae72425ef074680b605c96678ced123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70b4fccd75ba448291d4ce8bc00a6653" is_pre_election: true
I20260923 17:21:34.438007 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ea811d9af348cda1528b3deb3e8b2d" candidate_uuid: "cae72425ef074680b605c96678ced123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50971779fba844e9b00cb0605df42e35" is_pre_election: true
I20260923 17:21:34.438228 30871 raft_consensus.cc:2454] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae72425ef074680b605c96678ced123 in term 0.
I20260923 17:21:34.438613 31082 leader_election.cc:302] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50971779fba844e9b00cb0605df42e35, cae72425ef074680b605c96678ced123; no voters:
I20260923 17:21:34.438905 31295 raft_consensus.cc:2781] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:21:34.439035 31295 raft_consensus.cc:482] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:21:34.439172 31295 raft_consensus.cc:3037] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:34.440541 31321 raft_consensus.cc:504] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.441061 31010 raft_consensus.cc:2379] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cae72425ef074680b605c96678ced123 in current term 1: Already voted for candidate 70b4fccd75ba448291d4ce8bc00a6653 in this term.
I20260923 17:21:34.441614 31321 leader_election.cc:288] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [CANDIDATE]: Term 1 election: Requested vote from peers 50971779fba844e9b00cb0605df42e35 (127.28.72.193:41835), cae72425ef074680b605c96678ced123 (127.28.72.195:36001)
I20260923 17:21:34.441684 31295 raft_consensus.cc:504] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.441988 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ea811d9af348cda1528b3deb3e8b2d" candidate_uuid: "70b4fccd75ba448291d4ce8bc00a6653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50971779fba844e9b00cb0605df42e35"
I20260923 17:21:34.442210 30871 raft_consensus.cc:3037] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:34.442214 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ea811d9af348cda1528b3deb3e8b2d" candidate_uuid: "70b4fccd75ba448291d4ce8bc00a6653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae72425ef074680b605c96678ced123"
I20260923 17:21:34.442480 31153 raft_consensus.cc:2379] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 70b4fccd75ba448291d4ce8bc00a6653 in current term 1: Already voted for candidate cae72425ef074680b605c96678ced123 in this term.
I20260923 17:21:34.442894 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ea811d9af348cda1528b3deb3e8b2d" candidate_uuid: "cae72425ef074680b605c96678ced123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50971779fba844e9b00cb0605df42e35"
I20260923 17:21:34.443331 31295 leader_election.cc:288] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [CANDIDATE]: Term 1 election: Requested vote from peers 50971779fba844e9b00cb0605df42e35 (127.28.72.193:41835), 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194:34693)
I20260923 17:21:34.443692 31010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ea811d9af348cda1528b3deb3e8b2d" candidate_uuid: "cae72425ef074680b605c96678ced123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70b4fccd75ba448291d4ce8bc00a6653"
I20260923 17:21:34.443981 31010 raft_consensus.cc:2379] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cae72425ef074680b605c96678ced123 in current term 1: Already voted for candidate 70b4fccd75ba448291d4ce8bc00a6653 in this term.
I20260923 17:21:34.444453 31081 leader_election.cc:302] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cae72425ef074680b605c96678ced123; no voters: 50971779fba844e9b00cb0605df42e35, 70b4fccd75ba448291d4ce8bc00a6653
I20260923 17:21:34.444803 31295 raft_consensus.cc:2726] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260923 17:21:34.444999 30871 raft_consensus.cc:2454] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70b4fccd75ba448291d4ce8bc00a6653 in term 1.
I20260923 17:21:34.445482 30943 leader_election.cc:302] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50971779fba844e9b00cb0605df42e35, 70b4fccd75ba448291d4ce8bc00a6653; no voters: cae72425ef074680b605c96678ced123
I20260923 17:21:34.445791 31321 raft_consensus.cc:2781] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:34.446282 31321 raft_consensus.cc:686] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 LEADER]: Becoming Leader. State: Replica: 70b4fccd75ba448291d4ce8bc00a6653, State: Running, Role: LEADER
I20260923 17:21:34.446745 31321 consensus_queue.cc:237] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.450665 30712 catalog_manager.cc:5495] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194). New cstate: current_term: 1 leader_uuid: "70b4fccd75ba448291d4ce8bc00a6653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } health_report { overall_health: UNKNOWN } } }
I20260923 17:21:34.544938 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:34.538487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.28.72.194:34693)
0923 17:21:34.539057 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:34.539062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:34.539079 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:21:34.539218 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:34.539222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:34.539311 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:34.539459 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:34.539791 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.540641 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.542057 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.542594 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.542722 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:34.543321 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:34.544040 (+ 719us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:34.544170 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:34.544455 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 17:21:34.544562 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":333,"threads_started":1}
I20260923 17:21:34.563834 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:34.557899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:60393 (local address 127.28.72.193:41835)
0923 17:21:34.558067 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:34.558071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:34.558612 (+ 541us) server_negotiation.cc:407] Connection header received
0923 17:21:34.558769 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:34.558772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:34.558838 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:34.558936 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:34.559492 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.560188 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.561793 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.562217 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.562328 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:34.562919 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:34.563030 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:34.563190 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:34.563369 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 17:21:34.563528 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:21:34.563979 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:34.557786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:41835 (local address 127.28.72.194:60393)
0923 17:21:34.558515 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:34.558532 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:34.558659 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:34.558977 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:34.558980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:34.558989 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:21:34.559338 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:34.559347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.560437 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.560441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:34.561560 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:34.561574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.562483 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.562488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:34.562617 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:34.562682 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:21:34.563190 (+ 508us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:21:34.563206 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 17:21:34.563327 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 17:21:34.563544 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":408,"threads_started":1}
I20260923 17:21:34.573423 30871 raft_consensus.cc:1261] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Refusing update from remote peer 70b4fccd75ba448291d4ce8bc00a6653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:34.574193 31153 raft_consensus.cc:1261] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Refusing update from remote peer 70b4fccd75ba448291d4ce8bc00a6653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:34.574333 31326 consensus_queue.cc:1035] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:34.574887 31321 consensus_queue.cc:1035] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:34.610647 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:34.604203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:60085 (local address 127.28.72.194:34693)
0923 17:21:34.605018 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:34.605024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:34.605042 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:21:34.605111 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:34.605116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:34.605199 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:34.605273 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:34.605721 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.606523 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.608442 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.608988 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.609133 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:34.609724 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:34.609857 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:34.609938 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:34.610195 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 17:21:34.610283 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692}
I20260923 17:21:34.610690 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:34.604104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:34693 (local address 127.28.72.193:60085)
0923 17:21:34.604767 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:34.604783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:34.604934 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:34.605285 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:34.605289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:34.605298 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:21:34.605531 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:34.605539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.606945 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.606951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:34.608203 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:34.608217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:34.609198 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:34.609204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:34.609400 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:34.609499 (+ 99us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:21:34.610036 (+ 537us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:21:34.610047 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 17:21:34.610128 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:21:34.610247 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":327,"threads_started":1}
I20260923 17:21:34.642962 30712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:34.643253 30712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:34.646979 30712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c: Sending DeleteTablet for 3 replicas of tablet 79ea811d9af348cda1528b3deb3e8b2d
I20260923 17:21:34.647982 30850 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ea811d9af348cda1528b3deb3e8b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:34 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:58240
I20260923 17:21:34.648110 30989 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ea811d9af348cda1528b3deb3e8b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:34 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:50538
I20260923 17:21:34.648528 31130 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ea811d9af348cda1528b3deb3e8b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:34 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:46006
I20260923 17:21:34.651011 30712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:21:34.652064 30712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:21:34.659878 31350 tablet_replica.cc:330] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35: stopping tablet replica
I20260923 17:21:34.660115 31350 raft_consensus.cc:2229] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:34.660485 31350 raft_consensus.cc:2258] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:34.661657 31350 ts_tablet_manager.cc:1905] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:34.662540 31351 tablet_replica.cc:330] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653: stopping tablet replica
I20260923 17:21:34.662773 31351 raft_consensus.cc:2229] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:21:34.663565 31351 raft_consensus.cc:2258] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:34.664803 31351 ts_tablet_manager.cc:1905] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:34.664819 30850 tablet_service.cc:1467] Processing CreateTablet for tablet a898f5a503864f198e78e9cb39a6c7b5 (DEFAULT_TABLE table=test-table [id=8ddf216cdadc46208104c20e721c9f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:34.665297 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a898f5a503864f198e78e9cb39a6c7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:34.665278 30989 tablet_service.cc:1467] Processing CreateTablet for tablet a898f5a503864f198e78e9cb39a6c7b5 (DEFAULT_TABLE table=test-table [id=8ddf216cdadc46208104c20e721c9f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:34.665686 30989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a898f5a503864f198e78e9cb39a6c7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:34.665966 31129 tablet_service.cc:1467] Processing CreateTablet for tablet a898f5a503864f198e78e9cb39a6c7b5 (DEFAULT_TABLE table=test-table [id=8ddf216cdadc46208104c20e721c9f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:34.666384 31129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a898f5a503864f198e78e9cb39a6c7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:34.677785 31350 ts_tablet_manager.cc:1918] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:21:34.678005 31350 log.cc:1196] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/79ea811d9af348cda1528b3deb3e8b2d
I20260923 17:21:34.678532 31350 ts_tablet_manager.cc:1939] T 79ea811d9af348cda1528b3deb3e8b2d P 50971779fba844e9b00cb0605df42e35: Deleting consensus metadata
I20260923 17:21:34.678558 31352 tablet_replica.cc:330] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123: stopping tablet replica
I20260923 17:21:34.678793 31352 raft_consensus.cc:2229] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:34.679039 31352 raft_consensus.cc:2258] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:34.682045 31353 tablet_bootstrap.cc:492] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35: Bootstrap starting.
I20260923 17:21:34.682343 31354 tablet_bootstrap.cc:492] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653: Bootstrap starting.
I20260923 17:21:34.682351 31352 ts_tablet_manager.cc:1905] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:34.684954 31354 tablet_bootstrap.cc:654] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:34.682072 31355 tablet_bootstrap.cc:492] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123: Bootstrap starting.
I20260923 17:21:34.685920 30700 catalog_manager.cc:4841] TS 50971779fba844e9b00cb0605df42e35 (127.28.72.193:41835): tablet 79ea811d9af348cda1528b3deb3e8b2d (table test-table [id=434cb1e661d242bdb8cbb2a87403bc68]) successfully deleted
I20260923 17:21:34.680419 31351 ts_tablet_manager.cc:1918] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:21:34.686115 31351 log.cc:1196] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/79ea811d9af348cda1528b3deb3e8b2d
I20260923 17:21:34.686715 31353 tablet_bootstrap.cc:654] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:34.687079 31351 ts_tablet_manager.cc:1939] T 79ea811d9af348cda1528b3deb3e8b2d P 70b4fccd75ba448291d4ce8bc00a6653: Deleting consensus metadata
I20260923 17:21:34.687547 31355 tablet_bootstrap.cc:654] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:34.688329 30699 catalog_manager.cc:4841] TS 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194:34693): tablet 79ea811d9af348cda1528b3deb3e8b2d (table test-table [id=434cb1e661d242bdb8cbb2a87403bc68]) successfully deleted
I20260923 17:21:34.690729 31352 ts_tablet_manager.cc:1918] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:21:34.690927 31352 log.cc:1196] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/79ea811d9af348cda1528b3deb3e8b2d
I20260923 17:21:34.691548 31352 ts_tablet_manager.cc:1939] T 79ea811d9af348cda1528b3deb3e8b2d P cae72425ef074680b605c96678ced123: Deleting consensus metadata
I20260923 17:21:34.691684 31354 tablet_bootstrap.cc:492] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653: No bootstrap required, opened a new log
I20260923 17:21:34.691903 31354 ts_tablet_manager.cc:1397] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260923 17:21:34.692853 31354 raft_consensus.cc:348] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.693038 30699 catalog_manager.cc:4841] TS cae72425ef074680b605c96678ced123 (127.28.72.195:36001): tablet 79ea811d9af348cda1528b3deb3e8b2d (table test-table [id=434cb1e661d242bdb8cbb2a87403bc68]) successfully deleted
I20260923 17:21:34.693270 31354 raft_consensus.cc:374] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:34.693377 31354 raft_consensus.cc:729] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b4fccd75ba448291d4ce8bc00a6653, State: Initialized, Role: FOLLOWER
I20260923 17:21:34.693692 31354 consensus_queue.cc:260] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.694551 31354 ts_tablet_manager.cc:1428] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:21:34.695753 31353 tablet_bootstrap.cc:492] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35: No bootstrap required, opened a new log
I20260923 17:21:34.695922 31353 ts_tablet_manager.cc:1397] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
I20260923 17:21:34.696883 31353 raft_consensus.cc:348] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.697161 31353 raft_consensus.cc:374] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:34.697263 31353 raft_consensus.cc:729] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50971779fba844e9b00cb0605df42e35, State: Initialized, Role: FOLLOWER
I20260923 17:21:34.697553 31353 consensus_queue.cc:260] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.698096 31355 tablet_bootstrap.cc:492] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123: No bootstrap required, opened a new log
I20260923 17:21:34.698293 31355 ts_tablet_manager.cc:1397] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260923 17:21:34.699222 31355 raft_consensus.cc:348] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.699597 31355 raft_consensus.cc:374] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:34.699728 31355 raft_consensus.cc:729] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae72425ef074680b605c96678ced123, State: Initialized, Role: FOLLOWER
I20260923 17:21:34.699970 31355 consensus_queue.cc:260] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.700449 31353 ts_tablet_manager.cc:1428] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260923 17:21:34.700706 31355 ts_tablet_manager.cc:1428] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:21:34.775272 31347 raft_consensus.cc:482] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:21:34.775518 31347 raft_consensus.cc:504] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.776369 31347 leader_election.cc:288] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50971779fba844e9b00cb0605df42e35 (127.28.72.193:41835), cae72425ef074680b605c96678ced123 (127.28.72.195:36001)
I20260923 17:21:34.776960 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a898f5a503864f198e78e9cb39a6c7b5" candidate_uuid: "70b4fccd75ba448291d4ce8bc00a6653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50971779fba844e9b00cb0605df42e35" is_pre_election: true
I20260923 17:21:34.776960 31154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a898f5a503864f198e78e9cb39a6c7b5" candidate_uuid: "70b4fccd75ba448291d4ce8bc00a6653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae72425ef074680b605c96678ced123" is_pre_election: true
I20260923 17:21:34.777233 31154 raft_consensus.cc:2454] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70b4fccd75ba448291d4ce8bc00a6653 in term 0.
I20260923 17:21:34.777233 30871 raft_consensus.cc:2454] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70b4fccd75ba448291d4ce8bc00a6653 in term 0.
I20260923 17:21:34.777750 30943 leader_election.cc:302] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50971779fba844e9b00cb0605df42e35, 70b4fccd75ba448291d4ce8bc00a6653; no voters:
I20260923 17:21:34.778188 31347 raft_consensus.cc:2781] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:21:34.778365 31347 raft_consensus.cc:482] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:21:34.778499 31347 raft_consensus.cc:3037] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:34.781203 31347 raft_consensus.cc:504] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.781917 31347 leader_election.cc:288] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [CANDIDATE]: Term 1 election: Requested vote from peers 50971779fba844e9b00cb0605df42e35 (127.28.72.193:41835), cae72425ef074680b605c96678ced123 (127.28.72.195:36001)
I20260923 17:21:34.782682 31154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a898f5a503864f198e78e9cb39a6c7b5" candidate_uuid: "70b4fccd75ba448291d4ce8bc00a6653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae72425ef074680b605c96678ced123"
I20260923 17:21:34.782862 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a898f5a503864f198e78e9cb39a6c7b5" candidate_uuid: "70b4fccd75ba448291d4ce8bc00a6653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50971779fba844e9b00cb0605df42e35"
I20260923 17:21:34.782959 31154 raft_consensus.cc:3037] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:34.783099 30871 raft_consensus.cc:3037] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:34.786192 31154 raft_consensus.cc:2454] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70b4fccd75ba448291d4ce8bc00a6653 in term 1.
I20260923 17:21:34.786192 30871 raft_consensus.cc:2454] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70b4fccd75ba448291d4ce8bc00a6653 in term 1.
I20260923 17:21:34.786794 30942 leader_election.cc:302] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70b4fccd75ba448291d4ce8bc00a6653, cae72425ef074680b605c96678ced123; no voters:
I20260923 17:21:34.787253 31347 raft_consensus.cc:2781] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:34.787429 31347 raft_consensus.cc:686] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 LEADER]: Becoming Leader. State: Replica: 70b4fccd75ba448291d4ce8bc00a6653, State: Running, Role: LEADER
I20260923 17:21:34.788050 31347 consensus_queue.cc:237] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } }
I20260923 17:21:34.791785 30713 catalog_manager.cc:5495] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194). New cstate: current_term: 1 leader_uuid: "70b4fccd75ba448291d4ce8bc00a6653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70b4fccd75ba448291d4ce8bc00a6653" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 } health_report { overall_health: UNKNOWN } } }
I20260923 17:21:34.823171 31154 raft_consensus.cc:1261] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Refusing update from remote peer 70b4fccd75ba448291d4ce8bc00a6653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:34.823164 30871 raft_consensus.cc:1261] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Refusing update from remote peer 70b4fccd75ba448291d4ce8bc00a6653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:34.823915 31347 consensus_queue.cc:1035] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50971779fba844e9b00cb0605df42e35" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:34.824247 31343 consensus_queue.cc:1035] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae72425ef074680b605c96678ced123" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:21:34.860553 31331 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 17:21:36.519526 30712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:36.519781 30712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:36.524242 30712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1dcec500acf4607ba8d68d106a27a8c: Sending DeleteTablet for 3 replicas of tablet a898f5a503864f198e78e9cb39a6c7b5
I20260923 17:21:36.525144 30850 tablet_service.cc:1514] Processing DeleteTablet for tablet a898f5a503864f198e78e9cb39a6c7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:36 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:58240
I20260923 17:21:36.525242 30989 tablet_service.cc:1514] Processing DeleteTablet for tablet a898f5a503864f198e78e9cb39a6c7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:36 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:50538
I20260923 17:21:36.525372 31129 tablet_service.cc:1514] Processing DeleteTablet for tablet a898f5a503864f198e78e9cb39a6c7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:36 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:46006
I20260923 17:21:36.526062 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 30785
I20260923 17:21:36.536422 31389 tablet_replica.cc:330] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653: stopping tablet replica
I20260923 17:21:36.536929 31389 raft_consensus.cc:2229] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:21:36.537528 31389 raft_consensus.cc:2258] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:36.538666 31388 tablet_replica.cc:330] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35: stopping tablet replica
I20260923 17:21:36.538864 31389 ts_tablet_manager.cc:1905] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:36.538880 31388 raft_consensus.cc:2229] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:36.539301 31388 raft_consensus.cc:2258] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:36.541553 31388 ts_tablet_manager.cc:1905] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:36.543746 31390 tablet_replica.cc:330] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123: stopping tablet replica
I20260923 17:21:36.545372 31390 raft_consensus.cc:2229] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:36.545663 31390 raft_consensus.cc:2258] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:36.546623 31390 ts_tablet_manager.cc:1905] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:36.546815 31389 ts_tablet_manager.cc:1918] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:21:36.546944 31389 log.cc:1196] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/a898f5a503864f198e78e9cb39a6c7b5
I20260923 17:21:36.547412 31389 ts_tablet_manager.cc:1939] T a898f5a503864f198e78e9cb39a6c7b5 P 70b4fccd75ba448291d4ce8bc00a6653: Deleting consensus metadata
I20260923 17:21:36.549458 31388 ts_tablet_manager.cc:1918] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:21:36.549625 31388 log.cc:1196] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/a898f5a503864f198e78e9cb39a6c7b5
I20260923 17:21:36.549824 30699 catalog_manager.cc:4841] TS 70b4fccd75ba448291d4ce8bc00a6653 (127.28.72.194:34693): tablet a898f5a503864f198e78e9cb39a6c7b5 (table test-table [id=8ddf216cdadc46208104c20e721c9f94]) successfully deleted
I20260923 17:21:36.550138 31388 ts_tablet_manager.cc:1939] T a898f5a503864f198e78e9cb39a6c7b5 P 50971779fba844e9b00cb0605df42e35: Deleting consensus metadata
I20260923 17:21:36.551188 30700 catalog_manager.cc:4841] TS 50971779fba844e9b00cb0605df42e35 (127.28.72.193:41835): tablet a898f5a503864f198e78e9cb39a6c7b5 (table test-table [id=8ddf216cdadc46208104c20e721c9f94]) successfully deleted
I20260923 17:21:36.552500 31390 ts_tablet_manager.cc:1918] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:21:36.552644 31390 log.cc:1196] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/a898f5a503864f198e78e9cb39a6c7b5
I20260923 17:21:36.553054 31390 ts_tablet_manager.cc:1939] T a898f5a503864f198e78e9cb39a6c7b5 P cae72425ef074680b605c96678ced123: Deleting consensus metadata
I20260923 17:21:36.554045 30699 catalog_manager.cc:4841] TS cae72425ef074680b605c96678ced123 (127.28.72.195:36001): tablet a898f5a503864f198e78e9cb39a6c7b5 (table test-table [id=8ddf216cdadc46208104c20e721c9f94]) successfully deleted
Sep 23 17:21:36 dist-test-slave-j0m4 krb5kdc[30669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184094, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:21:36.566076 31392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:36.528110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48282 (local address 127.28.72.193:41835)
0923 17:21:36.540065 (+ 11955us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:36.540071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:36.540148 (+ 77us) server_negotiation.cc:407] Connection header received
0923 17:21:36.540213 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:36.540218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:36.540306 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:36.542281 (+ 1975us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:36.542474 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:36.544627 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:36.547249 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:36.547997 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:36.548736 (+ 739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:36.558952 (+ 10216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:36.558970 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:36.558986 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:36.559047 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:36.562435 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:36.563703 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:36.563711 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:36.563713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:36.563905 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:36.564335 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:36.564339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:36.564341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:36.564769 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:36.565014 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:36.565535 (+ 521us) server_negotiation.cc:299] Negotiation successful
0923 17:21:36.565758 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11797,"thread_start_us":15042,"threads_started":1}
W20260923 17:21:36.566982 30916 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48282
W20260923 17:21:36.567816 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 30785: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:21:36.568009 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 30785
I20260923 17:21:36.594985 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 30925
Sep 23 17:21:36 dist-test-slave-j0m4 krb5kdc[30669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184094, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:21:36.612001 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:36.595868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.28.72.194:34693)
0923 17:21:36.596463 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:36.596468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:36.596486 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:21:36.596543 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:36.596546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:36.596607 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:36.596757 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:36.598034 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:36.598945 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:36.600418 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:36.600884 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:36.600995 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:36.605598 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:36.605616 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:36.605632 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:36.605674 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:36.608813 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:36.609693 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:36.609701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:36.609704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:36.609850 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:36.610268 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:36.610272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:36.610274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:36.610703 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:36.610918 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:36.611426 (+ 508us) server_negotiation.cc:299] Negotiation successful
0923 17:21:36.611727 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":262,"threads_started":1}
W20260923 17:21:36.612802 31055 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35708
W20260923 17:21:36.613474 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 30925: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:21:36.613632 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 30925
I20260923 17:21:36.636602 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 31064
Sep 23 17:21:36 dist-test-slave-j0m4 krb5kdc[30669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184094, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:21:36.653466 31394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:36.637446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.28.72.195:36001)
0923 17:21:36.637964 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:36.637968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:36.637984 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:21:36.638036 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:36.638040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:36.638102 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:36.638194 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:36.639688 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:36.640453 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:36.641717 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:36.642106 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:36.642212 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:36.647512 (+ 5300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:36.647528 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:36.647545 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:36.647591 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:36.650684 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:36.651504 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:36.651512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:36.651514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:36.651651 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:36.652066 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:36.652069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:36.652071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:36.652421 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:36.652641 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:36.653002 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 17:21:36.653182 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":303,"threads_started":1}
W20260923 17:21:36.654177 31195 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42980
W20260923 17:21:36.654852 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 31064: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:21:36.654999 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 31064
I20260923 17:21:36.677117 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 30682
I20260923 17:21:36.678290 30754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:36.799615 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 30682
2026-09-23T17:21:36Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9386 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:21:36 dist-test-slave-j0m4 krb5kdc[31398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:21:36 dist-test-slave-j0m4 krb5kdc[31398](info): set up 2 sockets
Sep 23 17:21:36 dist-test-slave-j0m4 krb5kdc[31398](info): commencing operation
krb5kdc: starting...
W20260923 17:21:37.923708 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:21:38 dist-test-slave-j0m4 krb5kdc[31398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:21:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:21:38.135896 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:39969
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:39969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:38.477846 31411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:38.478204 31411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:38.478425 31411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:38.486090 31411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:38.486215 31411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:38.486294 31411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:38.486387 31411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:38.486465 31411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:21:38.497186 31411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:39969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:39969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:38.498672 31411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:38.500229 31411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:38.513468 31419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:38.513525 31417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:38.513895 31416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:38.515085 31411 server_base.cc:1034] running on GCE node
I20260923 17:21:38.516124 31411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:38.517872 31411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:38.519162 31411 hybrid_clock.cc:648] HybridClock initialized: now 1790184098519056 us; error 43 us; skew 500 ppm
I20260923 17:21:38.519676 31411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:38 dist-test-slave-j0m4 krb5kdc[31398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:38.523494 31411 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:21:38.525172 31411 webserver.cc:460] Webserver started at http://127.28.72.254:46735/ using document root <none> and password file <none>
I20260923 17:21:38.525905 31411 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:38.526011 31411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:38.526278 31411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:38.529486 31411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "9d0ff13dc1fb4ee88737181421a645e6"
format_stamp: "Formatted at 2026-09-23 17:21:38 on dist-test-slave-j0m4"
server_key: "0c477e239fdec7c01fa2498f16c94460"
server_key_iv: "09d923f0625ad34b5bb5e9145f05967e"
server_key_version: "encryptionkey@0"
I20260923 17:21:38.530238 31411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "9d0ff13dc1fb4ee88737181421a645e6"
format_stamp: "Formatted at 2026-09-23 17:21:38 on dist-test-slave-j0m4"
server_key: "0c477e239fdec7c01fa2498f16c94460"
server_key_iv: "09d923f0625ad34b5bb5e9145f05967e"
server_key_version: "encryptionkey@0"
I20260923 17:21:38.536017 31411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:21:38.540230 31426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:38.542093 31411 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:21:38.542343 31411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "9d0ff13dc1fb4ee88737181421a645e6"
format_stamp: "Formatted at 2026-09-23 17:21:38 on dist-test-slave-j0m4"
server_key: "0c477e239fdec7c01fa2498f16c94460"
server_key_iv: "09d923f0625ad34b5bb5e9145f05967e"
server_key_version: "encryptionkey@0"
I20260923 17:21:38.542650 31411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:38.563831 31411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:38.569027 31411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:38.569372 31411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:38.589437 31411 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:39969
I20260923 17:21:38.589457 31477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:39969 every 8 connection(s)
I20260923 17:21:38.591046 31411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:38.595322 31478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:38.597067 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 31411
I20260923 17:21:38.597281 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:21:38.598101 28963 external_mini_cluster.cc:1428] Setting key 266d5409b5f4edea358863a53ce36e4a
I20260923 17:21:38.610756 31478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6: Bootstrap starting.
Sep 23 17:21:38 dist-test-slave-j0m4 krb5kdc[31398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:38.614073 31478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:38.615381 31478 log.cc:824] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:38.618338 31478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6: No bootstrap required, opened a new log
I20260923 17:21:38.621901 31481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:38.600429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.28.72.254:39969)
0923 17:21:38.601284 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:38.601297 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:38.601350 (+ 53us) server_negotiation.cc:407] Connection header received
0923 17:21:38.603018 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:38.603044 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:38.603578 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:38.603942 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:38.605849 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:38.607365 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:38.609261 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:38.609989 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:38.610141 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:38.614365 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:38.614374 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:38.614392 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:38.614445 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:38.617607 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:38.618359 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:38.618366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:38.618373 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:38.618590 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:38.618973 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:38.618978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:38.618982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:38.619604 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:38.619922 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:38.620404 (+ 482us) server_negotiation.cc:299] Negotiation successful
0923 17:21:38.620840 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":401,"threads_started":1}
I20260923 17:21:38.627213 31478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d0ff13dc1fb4ee88737181421a645e6" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 39969 } }
I20260923 17:21:38.627640 31478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:38.627753 31478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0ff13dc1fb4ee88737181421a645e6, State: Initialized, Role: FOLLOWER
I20260923 17:21:38.628392 31478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d0ff13dc1fb4ee88737181421a645e6" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 39969 } }
I20260923 17:21:38.628657 31478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:38.628803 31478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:38.628974 31478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:38.631560 31478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d0ff13dc1fb4ee88737181421a645e6" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 39969 } }
I20260923 17:21:38.632103 31478 leader_election.cc:302] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d0ff13dc1fb4ee88737181421a645e6; no voters:
I20260923 17:21:38.632884 31478 leader_election.cc:288] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:38.633049 31483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:38.633802 31483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [term 1 LEADER]: Becoming Leader. State: Replica: 9d0ff13dc1fb4ee88737181421a645e6, State: Running, Role: LEADER
I20260923 17:21:38.634346 31483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d0ff13dc1fb4ee88737181421a645e6" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 39969 } }
I20260923 17:21:38.635061 31478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:38.637727 31485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d0ff13dc1fb4ee88737181421a645e6. Latest consensus state: current_term: 1 leader_uuid: "9d0ff13dc1fb4ee88737181421a645e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d0ff13dc1fb4ee88737181421a645e6" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 39969 } } }
I20260923 17:21:38.637880 31484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d0ff13dc1fb4ee88737181421a645e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d0ff13dc1fb4ee88737181421a645e6" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 39969 } } }
I20260923 17:21:38.638167 31484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:38.638167 31485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6 [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:38.640967 31490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:38.646726 31490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:38.656009 31490 catalog_manager.cc:1347] Generated new cluster ID: 18484f426f0a4d9b9d62a5734d51bdc9
I20260923 17:21:38.656152 31490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:38.674563 31490 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:38.676028 31490 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:38.687541 31490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d0ff13dc1fb4ee88737181421a645e6: Generated new TSK 0
I20260923 17:21:38.688550 31490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:21:38.751617 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:39969
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:39.115523 31504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:39.115847 31504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:39.116039 31504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:39.124191 31504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:39.124313 31504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:39.124502 31504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:21:39.134948 31504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:39969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:39.136667 31504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:39.138044 31504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:39.153198 31510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:39.153291 31512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:39.153637 31509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:39.154490 31504 server_base.cc:1034] running on GCE node
I20260923 17:21:39.155226 31504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:39.156230 31504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:39.157471 31504 hybrid_clock.cc:648] HybridClock initialized: now 1790184099157410 us; error 51 us; skew 500 ppm
I20260923 17:21:39.157789 31504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:39 dist-test-slave-j0m4 krb5kdc[31398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184099, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:39.161243 31504 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:21:39.162784 31504 webserver.cc:460] Webserver started at http://127.28.72.193:33421/ using document root <none> and password file <none>
I20260923 17:21:39.163554 31504 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:39.163694 31504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:39.164019 31504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:39.167022 31504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "1872a53e438d41b19f0a08fffc7022ca"
format_stamp: "Formatted at 2026-09-23 17:21:39 on dist-test-slave-j0m4"
server_key: "877af42920b3c1b9380c0c1e7f873453"
server_key_iv: "5741d4fbf1346e3c62d6b1bfee2bb088"
server_key_version: "encryptionkey@0"
I20260923 17:21:39.167913 31504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "1872a53e438d41b19f0a08fffc7022ca"
format_stamp: "Formatted at 2026-09-23 17:21:39 on dist-test-slave-j0m4"
server_key: "877af42920b3c1b9380c0c1e7f873453"
server_key_iv: "5741d4fbf1346e3c62d6b1bfee2bb088"
server_key_version: "encryptionkey@0"
I20260923 17:21:39.173316 31504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:21:39.177142 31519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:39.178578 31504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:21:39.178776 31504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "1872a53e438d41b19f0a08fffc7022ca"
format_stamp: "Formatted at 2026-09-23 17:21:39 on dist-test-slave-j0m4"
server_key: "877af42920b3c1b9380c0c1e7f873453"
server_key_iv: "5741d4fbf1346e3c62d6b1bfee2bb088"
server_key_version: "encryptionkey@0"
I20260923 17:21:39.179032 31504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:39.193002 31504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:39.198488 31504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:39.198841 31504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:39.199998 31504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:39.201488 31504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:39.201596 31504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:39.201745 31504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:39.201817 31504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:39.237775 31504 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:44015
I20260923 17:21:39.237867 31631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:44015 every 8 connection(s)
I20260923 17:21:39.239094 31504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:21:39.245066 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 31504
I20260923 17:21:39.245270 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:21:39.245962 28963 external_mini_cluster.cc:1428] Setting key ad50de030a99eb931226263455ad1e79
Sep 23 17:21:39 dist-test-slave-j0m4 krb5kdc[31398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184099, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:39.260721 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:39.242090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:57911 (local address 127.28.72.254:39969)
0923 17:21:39.242607 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:39.242611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:39.243695 (+ 1084us) server_negotiation.cc:407] Connection header received
0923 17:21:39.245630 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:39.245635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:39.245723 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:39.245855 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:39.247653 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:39.248487 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:39.250133 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:39.250551 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:39.250695 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:39.254557 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:39.254563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:39.254566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:39.254597 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:39.257217 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:39.258147 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:39.258159 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:39.258161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:39.258304 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:39.258780 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:39.258784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:39.258788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:39.259088 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:39.259310 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:39.260065 (+ 755us) server_negotiation.cc:299] Negotiation successful
0923 17:21:39.260386 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":247,"threads_started":1}
I20260923 17:21:39.261710 31635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:39.242639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:39969 (local address 127.28.72.193:57911)
0923 17:21:39.243518 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:39.243565 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:39.245335 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:39.246125 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:39.246136 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:39.246744 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:39.247462 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:39.247481 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:39.248655 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:39.248661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:39.249980 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:39.249992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:39.250787 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:39.250795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:39.251047 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:39.251810 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:39.251834 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:39.254314 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:39.257372 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:39.257381 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:39.257398 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:39.257890 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:39.258427 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:39.258432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:39.258436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:39.258638 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:39.259278 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:39.259287 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:39.259672 (+ 385us) client_negotiation.cc:769] Sending connection context
0923 17:21:39.259955 (+ 283us) client_negotiation.cc:240] Negotiation successful
0923 17:21:39.260526 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":291,"threads_started":1}
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
I20260923 17:21:39.263541 31632 heartbeater.cc:347] Connected to a master server at 127.28.72.254:39969
I20260923 17:21:39.263821 31632 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:39.264568 31632 heartbeater.cc:511] Master 127.28.72.254:39969 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
I20260923 17:21:39.267369 31443 ts_manager.cc:194] Registered new tserver with Master: 1872a53e438d41b19f0a08fffc7022ca (127.28.72.193:44015)
I20260923 17:21:39.268904 31443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:57911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:21:39.296588 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:39969
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:39.632292 31638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:39.632598 31638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:39.632756 31638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:39.640496 31638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:39.640664 31638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:39.640841 31638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:21:39.652465 31638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:39969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:39.653810 31638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:39.655184 31638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:39.669489 31643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:39.669521 31646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:39.669450 31644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:39.669740 31638 server_base.cc:1034] running on GCE node
I20260923 17:21:39.670784 31638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:39.671904 31638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:39.673162 31638 hybrid_clock.cc:648] HybridClock initialized: now 1790184099673114 us; error 48 us; skew 500 ppm
I20260923 17:21:39.673517 31638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:39 dist-test-slave-j0m4 krb5kdc[31398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184099, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:39.677273 31638 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:21:39.678967 31638 webserver.cc:460] Webserver started at http://127.28.72.194:40685/ using document root <none> and password file <none>
I20260923 17:21:39.679791 31638 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:39.679958 31638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:39.680277 31638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:39.683694 31638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "073eec80fe314245af80163dddf4cd50"
format_stamp: "Formatted at 2026-09-23 17:21:39 on dist-test-slave-j0m4"
server_key: "c213535a3fe6b310dc976188b6fc43be"
server_key_iv: "699a3a38ac726cc0387f455b823cbcdf"
server_key_version: "encryptionkey@0"
I20260923 17:21:39.684603 31638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "073eec80fe314245af80163dddf4cd50"
format_stamp: "Formatted at 2026-09-23 17:21:39 on dist-test-slave-j0m4"
server_key: "c213535a3fe6b310dc976188b6fc43be"
server_key_iv: "699a3a38ac726cc0387f455b823cbcdf"
server_key_version: "encryptionkey@0"
I20260923 17:21:39.690860 31638 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 17:21:39.695180 31653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:39.697026 31638 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:21:39.697249 31638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "073eec80fe314245af80163dddf4cd50"
format_stamp: "Formatted at 2026-09-23 17:21:39 on dist-test-slave-j0m4"
server_key: "c213535a3fe6b310dc976188b6fc43be"
server_key_iv: "699a3a38ac726cc0387f455b823cbcdf"
server_key_version: "encryptionkey@0"
I20260923 17:21:39.697486 31638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:39.715682 31638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:39.720347 31638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:39.720654 31638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:39.721692 31638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:39.723096 31638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:39.723255 31638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:39.723410 31638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:39.723483 31638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:39.760936 31638 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:35287
I20260923 17:21:39.760994 31765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:35287 every 8 connection(s)
I20260923 17:21:39.762493 31638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:21:39.767446 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 31638
I20260923 17:21:39.767707 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:21:39.768137 28963 external_mini_cluster.cc:1428] Setting key e839797015cc993af6bd4ba29cd66994
Sep 23 17:21:39 dist-test-slave-j0m4 krb5kdc[31398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184099, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
I20260923 17:21:39.795814 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:39.765158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:50625 (local address 127.28.72.254:39969)
0923 17:21:39.765607 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:39.765613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:39.766858 (+ 1245us) server_negotiation.cc:407] Connection header received
0923 17:21:39.768883 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:39.768895 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:39.768965 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:39.769054 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:39.772426 (+ 3372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:39.773722 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:39.776888 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:39.777635 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:39.777800 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:39.784602 (+ 6802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:39.784611 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:39.784617 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:39.784668 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:39.788842 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:39.790131 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:39.790140 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:39.790145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:39.790319 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:39.790873 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:39.790882 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:39.790887 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:39.791362 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:39.791533 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:39.795065 (+ 3532us) server_negotiation.cc:299] Negotiation successful
0923 17:21:39.795389 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":211,"threads_started":1}
I20260923 17:21:39.796319 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:39.765866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:39969 (local address 127.28.72.194:50625)
0923 17:21:39.766681 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:39.766721 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:39.768602 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:39.769314 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:39.769323 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:39.771002 (+ 1679us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:39.772129 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:39.772212 (+ 83us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:39.773912 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:39.773946 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:39.776688 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:39.776708 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:39.778136 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:39.778146 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:39.778521 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:39.779393 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:39.779427 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:39.784199 (+ 4772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:39.789065 (+ 4866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:39.789077 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:39.789101 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:39.789917 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:39.790447 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:39.790453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:39.790456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:39.790738 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:39.791621 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:39.791636 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:39.792193 (+ 557us) client_negotiation.cc:769] Sending connection context
0923 17:21:39.792565 (+ 372us) client_negotiation.cc:240] Negotiation successful
0923 17:21:39.793283 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":296,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
I20260923 17:21:39.798651 31766 heartbeater.cc:347] Connected to a master server at 127.28.72.254:39969
I20260923 17:21:39.799075 31766 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:39.800391 31766 heartbeater.cc:511] Master 127.28.72.254:39969 requested a full tablet report, sending...
I20260923 17:21:39.803061 31443 ts_manager.cc:194] Registered new tserver with Master: 073eec80fe314245af80163dddf4cd50 (127.28.72.194:35287)
I20260923 17:21:39.804603 31443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:50625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:21:39.831964 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:39969
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:40.180335 31772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:40.180786 31772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:40.180970 31772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:40.189239 31772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:40.189427 31772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:40.189615 31772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:21:40.200253 31772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:39969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:40.202028 31772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:40.203560 31772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:40.217391 31777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:40.217614 31778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:40.217612 31780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:40.218000 31772 server_base.cc:1034] running on GCE node
I20260923 17:21:40.218811 31772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:40.220091 31772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:40.221338 31772 hybrid_clock.cc:648] HybridClock initialized: now 1790184100221290 us; error 45 us; skew 500 ppm
I20260923 17:21:40.221714 31772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:40 dist-test-slave-j0m4 krb5kdc[31398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:40.225086 31772 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:21:40.226639 31772 webserver.cc:460] Webserver started at http://127.28.72.195:40663/ using document root <none> and password file <none>
I20260923 17:21:40.227497 31772 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:40.227640 31772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:40.227969 31772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:40.230950 31772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "3a71354fc49341c98b2bd8287650076b"
format_stamp: "Formatted at 2026-09-23 17:21:40 on dist-test-slave-j0m4"
server_key: "073b17de8d1714fb473e7469281dd103"
server_key_iv: "79fd4f12a2f0b0f429b7d0127a1274a6"
server_key_version: "encryptionkey@0"
I20260923 17:21:40.231834 31772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "3a71354fc49341c98b2bd8287650076b"
format_stamp: "Formatted at 2026-09-23 17:21:40 on dist-test-slave-j0m4"
server_key: "073b17de8d1714fb473e7469281dd103"
server_key_iv: "79fd4f12a2f0b0f429b7d0127a1274a6"
server_key_version: "encryptionkey@0"
I20260923 17:21:40.237589 31772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:21:40.242067 31787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:40.243820 31772 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 17:21:40.244026 31772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "3a71354fc49341c98b2bd8287650076b"
format_stamp: "Formatted at 2026-09-23 17:21:40 on dist-test-slave-j0m4"
server_key: "073b17de8d1714fb473e7469281dd103"
server_key_iv: "79fd4f12a2f0b0f429b7d0127a1274a6"
server_key_version: "encryptionkey@0"
I20260923 17:21:40.244247 31772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:40.258986 31772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:40.264318 31772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:40.264639 31772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:40.265733 31772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:40.267309 31772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:40.267462 31772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:40.267607 31772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:40.267685 31772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:40.272827 31632 heartbeater.cc:503] Master 127.28.72.254:39969 was elected leader, sending a full tablet report...
I20260923 17:21:40.304029 31772 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:42083
I20260923 17:21:40.304101 31899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:42083 every 8 connection(s)
I20260923 17:21:40.305485 31772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:21:40.313422 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 31772
I20260923 17:21:40.313642 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:21:40.314095 28963 external_mini_cluster.cc:1428] Setting key 2d113df4a73d3ed16d145e430237fb29
Sep 23 17:21:40 dist-test-slave-j0m4 krb5kdc[31398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:40.327723 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:40.308364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:56683 (local address 127.28.72.254:39969)
0923 17:21:40.308838 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:40.308845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:40.309977 (+ 1132us) server_negotiation.cc:407] Connection header received
0923 17:21:40.311849 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:40.311854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:40.311949 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:40.312055 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:40.313749 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:40.314631 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:40.316974 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:40.317656 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:40.317780 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:40.321479 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:40.321484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:40.321487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:40.321522 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:40.324293 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:40.325183 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:40.325187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:40.325190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:40.325323 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:40.325838 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:40.325844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:40.325846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:40.326148 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:40.326274 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:40.327221 (+ 947us) server_negotiation.cc:299] Negotiation successful
0923 17:21:40.327452 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":211,"threads_started":1}
I20260923 17:21:40.329370 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:40.308986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:39969 (local address 127.28.72.195:56683)
0923 17:21:40.309764 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:40.309803 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:40.311557 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:40.312281 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:40.312290 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:40.312895 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:40.313486 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:40.313505 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:40.314822 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:40.314826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:40.316808 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:40.316822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:40.317956 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:40.317962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:40.318182 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:40.318880 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:40.318927 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:40.321252 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:40.324472 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:40.324481 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:40.324495 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:40.325012 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:40.325493 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:40.325496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:40.325499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:40.325696 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:40.326385 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:40.326391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:40.326749 (+ 358us) client_negotiation.cc:769] Sending connection context
0923 17:21:40.327016 (+ 267us) client_negotiation.cc:240] Negotiation successful
0923 17:21:40.327591 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":374,"threads_started":1}
I20260923 17:21:40.330931 31900 heartbeater.cc:347] Connected to a master server at 127.28.72.254:39969
I20260923 17:21:40.331301 31900 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:40.332098 31900 heartbeater.cc:511] Master 127.28.72.254:39969 requested a full tablet report, sending...
I20260923 17:21:40.333932 31442 ts_manager.cc:194] Registered new tserver with Master: 3a71354fc49341c98b2bd8287650076b (127.28.72.195:42083)
I20260923 17:21:40.334864 31442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:56683
I20260923 17:21:40.343406 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 17:21:40.376120 31902 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:21:40.373529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.28.72.254:39969)
0923 17:21:40.373746 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:40.373751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:40.374183 (+ 432us) server_negotiation.cc:407] Connection header received
0923 17:21:40.374440 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:40.374444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:40.374511 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:40.374615 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:40.375935 (+ 1320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60308: BlockingRecv error: recv got EOF from 127.0.0.1:60308 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 17:21:40.379957 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 31504
W20260923 17:21:40.383267 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 31504: Not authorized: Client connection negotiation failed: client connection to 127.28.72.193:44015: server requires authentication, but client does not have Kerberos credentials available
I20260923 17:21:40.383502 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 31504
I20260923 17:21:40.399353 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 31638
W20260923 17:21:40.402493 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 31638: Not authorized: Client connection negotiation failed: client connection to 127.28.72.194:35287: server requires authentication, but client does not have Kerberos credentials available
I20260923 17:21:40.402671 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 31638
W20260923 17:21:40.402809 31913 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:21:40.400396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.28.72.194:35287)
0923 17:21:40.401084 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:40.401092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:40.401119 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:21:40.401180 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:40.401187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:40.401435 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:40.401576 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:40.402427 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49916: BlockingRecv error: recv got EOF from 127.0.0.1:49916 (error 108)
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":350,"threads_started":1}
I20260923 17:21:40.418929 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 31772
W20260923 17:21:40.422091 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 31772: Not authorized: Client connection negotiation failed: client connection to 127.28.72.195:42083: server requires authentication, but client does not have Kerberos credentials available
I20260923 17:21:40.422277 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 31772
I20260923 17:21:40.438165 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 31411
I20260923 17:21:40.439601 31473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:40.546489 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 31411
2026-09-23T17:21:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3745 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:21:40 dist-test-slave-j0m4 krb5kdc[31918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:21:40 dist-test-slave-j0m4 krb5kdc[31918](info): set up 2 sockets
Sep 23 17:21:40 dist-test-slave-j0m4 krb5kdc[31918](info): commencing operation
krb5kdc: starting...
W20260923 17:21:41.683828 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:21:41 dist-test-slave-j0m4 krb5kdc[31918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:21:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:21:41.906258 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:43821
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:42.272814 31931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:42.273137 31931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:42.273293 31931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:42.280875 31931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:42.281002 31931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:42.281086 31931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:42.281176 31931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:42.281260 31931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:21:42.291579 31931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:43821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:42.292955 31931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:42.294312 31931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:42.306928 31937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:42.307248 31936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:42.307572 31939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:42.308027 31931 server_base.cc:1034] running on GCE node
I20260923 17:21:42.308955 31931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:42.310474 31931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:42.311724 31931 hybrid_clock.cc:648] HybridClock initialized: now 1790184102311679 us; error 38 us; skew 500 ppm
I20260923 17:21:42.312129 31931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:42 dist-test-slave-j0m4 krb5kdc[31918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184102, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:42.325537 31931 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:21:42.327394 31931 webserver.cc:460] Webserver started at http://127.28.72.254:36785/ using document root <none> and password file <none>
I20260923 17:21:42.328169 31931 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:42.328328 31931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:42.328660 31931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:42.331915 31931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "f129153b1dcc4a9d82c23bd133769c6b"
format_stamp: "Formatted at 2026-09-23 17:21:42 on dist-test-slave-j0m4"
server_key: "324ec01d2d3bbc02acf9843cbf89c1c4"
server_key_iv: "4e2c228b370c753966376aa20f25aea0"
server_key_version: "encryptionkey@0"
I20260923 17:21:42.333163 31931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "f129153b1dcc4a9d82c23bd133769c6b"
format_stamp: "Formatted at 2026-09-23 17:21:42 on dist-test-slave-j0m4"
server_key: "324ec01d2d3bbc02acf9843cbf89c1c4"
server_key_iv: "4e2c228b370c753966376aa20f25aea0"
server_key_version: "encryptionkey@0"
I20260923 17:21:42.339051 31931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:21:42.342955 31946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:42.344621 31931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:21:42.344842 31931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "f129153b1dcc4a9d82c23bd133769c6b"
format_stamp: "Formatted at 2026-09-23 17:21:42 on dist-test-slave-j0m4"
server_key: "324ec01d2d3bbc02acf9843cbf89c1c4"
server_key_iv: "4e2c228b370c753966376aa20f25aea0"
server_key_version: "encryptionkey@0"
I20260923 17:21:42.345070 31931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:42.360077 31931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:42.364923 31931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:42.365247 31931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:42.384882 31931 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:43821
I20260923 17:21:42.384932 31997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:43821 every 8 connection(s)
I20260923 17:21:42.386345 31931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:42.389094 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 31931
I20260923 17:21:42.389351 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:21:42.390118 28963 external_mini_cluster.cc:1428] Setting key 1864ea370711962886d3ae1695a3ebee
I20260923 17:21:42.391022 31998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:21:42 dist-test-slave-j0m4 krb5kdc[31918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:42.406451 31998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b: Bootstrap starting.
I20260923 17:21:42.410176 31998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:42.411526 31998 log.cc:824] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:42.414501 31998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b: No bootstrap required, opened a new log
I20260923 17:21:42.414534 32001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:42.392100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.28.72.254:43821)
0923 17:21:42.393002 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:42.393018 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:42.393060 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:21:42.394647 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:42.394671 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:42.395177 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:42.395577 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:42.397493 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:42.398971 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:42.400759 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:42.401482 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:42.401664 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:42.406628 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:42.406641 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:42.406668 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:42.406717 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:42.409860 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:42.410836 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:42.410845 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:42.410854 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:42.411053 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:42.411576 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:42.411579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:42.411581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:42.412204 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:42.412540 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:42.413002 (+ 462us) server_negotiation.cc:299] Negotiation successful
0923 17:21:42.413435 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":446,"threads_started":1}
I20260923 17:21:42.418928 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:21:42.422878 31998 raft_consensus.cc:348] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f129153b1dcc4a9d82c23bd133769c6b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43821 } }
I20260923 17:21:42.423369 31998 raft_consensus.cc:374] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:42.423492 31998 raft_consensus.cc:729] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f129153b1dcc4a9d82c23bd133769c6b, State: Initialized, Role: FOLLOWER
I20260923 17:21:42.424072 31998 consensus_queue.cc:260] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f129153b1dcc4a9d82c23bd133769c6b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43821 } }
I20260923 17:21:42.424316 31998 raft_consensus.cc:388] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:42.424470 31998 raft_consensus.cc:482] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:42.424654 31998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:42.427779 31998 raft_consensus.cc:504] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f129153b1dcc4a9d82c23bd133769c6b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43821 } }
I20260923 17:21:42.428401 31998 leader_election.cc:302] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f129153b1dcc4a9d82c23bd133769c6b; no voters:
I20260923 17:21:42.429105 31998 leader_election.cc:288] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:42.429235 32004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:42.430562 32004 raft_consensus.cc:686] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [term 1 LEADER]: Becoming Leader. State: Replica: f129153b1dcc4a9d82c23bd133769c6b, State: Running, Role: LEADER
I20260923 17:21:42.431192 32004 consensus_queue.cc:237] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f129153b1dcc4a9d82c23bd133769c6b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43821 } }
I20260923 17:21:42.431582 31998 sys_catalog.cc:564] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:42.434122 32005 sys_catalog.cc:455] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f129153b1dcc4a9d82c23bd133769c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f129153b1dcc4a9d82c23bd133769c6b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43821 } } }
I20260923 17:21:42.434062 32006 sys_catalog.cc:455] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f129153b1dcc4a9d82c23bd133769c6b. Latest consensus state: current_term: 1 leader_uuid: "f129153b1dcc4a9d82c23bd133769c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f129153b1dcc4a9d82c23bd133769c6b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43821 } } }
I20260923 17:21:42.434474 32006 sys_catalog.cc:458] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:42.434474 32005 sys_catalog.cc:458] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:42.438349 32010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:42.446949 32010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:42.476972 32010 catalog_manager.cc:1347] Generated new cluster ID: 1db24e5daa474fa9959629ed74d1f054
I20260923 17:21:42.477108 32010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:42.489985 32010 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:42.492164 32010 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:42.506551 32010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f129153b1dcc4a9d82c23bd133769c6b: Generated new TSK 0
I20260923 17:21:42.507649 32010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:21:42.510842 32001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:42.501306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.28.72.254:43821)
0923 17:21:42.501483 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:42.501490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:42.502089 (+ 599us) server_negotiation.cc:407] Connection header received
0923 17:21:42.502241 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:42.502247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:42.502339 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:42.502463 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:42.503858 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:42.505033 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:42.506985 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:42.507705 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:42.507845 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:42.509331 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:42.509335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:42.509339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:42.510022 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:42.510069 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:42.510098 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:42.510189 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:42.510364 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 17:21:42.510452 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:21:42.519696 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 31931
I20260923 17:21:42.521183 31993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:42.612802 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 31931
2026-09-23T17:21:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2055 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:21:42 dist-test-slave-j0m4 krb5kdc[32032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:21:42 dist-test-slave-j0m4 krb5kdc[32032](info): set up 2 sockets
Sep 23 17:21:42 dist-test-slave-j0m4 krb5kdc[32032](info): commencing operation
krb5kdc: starting...
W20260923 17:21:43.700119 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.012s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:21:43 dist-test-slave-j0m4 krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:21:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:21:43.920536 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:44289
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:44289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:44.267644 32045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:44.267970 32045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:44.268138 32045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:44.275906 32045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:44.276041 32045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:44.276135 32045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:44.276237 32045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:44.276321 32045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:21:44.286975 32045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:44289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:44289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:44.288479 32045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:44.289851 32045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:44.302757 32050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:44.302706 32051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:44.302778 32053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:44.303375 32045 server_base.cc:1034] running on GCE node
I20260923 17:21:44.304239 32045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:44.305760 32045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:44.306996 32045 hybrid_clock.cc:648] HybridClock initialized: now 1790184104306934 us; error 55 us; skew 500 ppm
I20260923 17:21:44.307384 32045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:44 dist-test-slave-j0m4 krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184104, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:44.310897 32045 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:21:44.312578 32045 webserver.cc:460] Webserver started at http://127.28.72.254:41247/ using document root <none> and password file <none>
I20260923 17:21:44.313303 32045 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:44.313453 32045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:44.313778 32045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:44.316972 32045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "790e9dabcff84cdc9bb80d8a265d35cc"
format_stamp: "Formatted at 2026-09-23 17:21:44 on dist-test-slave-j0m4"
server_key: "3153058c8a6353c7641a051fa9aff907"
server_key_iv: "88c6725c2ada5ecd1dcf10984aff52e6"
server_key_version: "encryptionkey@0"
I20260923 17:21:44.317795 32045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "790e9dabcff84cdc9bb80d8a265d35cc"
format_stamp: "Formatted at 2026-09-23 17:21:44 on dist-test-slave-j0m4"
server_key: "3153058c8a6353c7641a051fa9aff907"
server_key_iv: "88c6725c2ada5ecd1dcf10984aff52e6"
server_key_version: "encryptionkey@0"
I20260923 17:21:44.323812 32045 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 17:21:44.328075 32060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:44.329789 32045 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:21:44.330049 32045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "790e9dabcff84cdc9bb80d8a265d35cc"
format_stamp: "Formatted at 2026-09-23 17:21:44 on dist-test-slave-j0m4"
server_key: "3153058c8a6353c7641a051fa9aff907"
server_key_iv: "88c6725c2ada5ecd1dcf10984aff52e6"
server_key_version: "encryptionkey@0"
I20260923 17:21:44.330250 32045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:44.347234 32045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:44.352033 32045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:44.352355 32045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:44.371691 32045 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:44289
I20260923 17:21:44.371726 32111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:44289 every 8 connection(s)
I20260923 17:21:44.373185 32045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:44.377259 32112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:44.382534 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 32045
I20260923 17:21:44.382786 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:21:44.383463 28963 external_mini_cluster.cc:1428] Setting key 1b792fa6a04979ed4e302f358385d32d
I20260923 17:21:44.389561 32112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc: Bootstrap starting.
I20260923 17:21:44.393637 32112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:44.394747 32112 log.cc:824] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:44.397980 32112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc: No bootstrap required, opened a new log
Sep 23 17:21:44 dist-test-slave-j0m4 krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:44.408610 32112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790e9dabcff84cdc9bb80d8a265d35cc" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44289 } }
I20260923 17:21:44.409180 32112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:44.409309 32112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 790e9dabcff84cdc9bb80d8a265d35cc, State: Initialized, Role: FOLLOWER
I20260923 17:21:44.410028 32112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790e9dabcff84cdc9bb80d8a265d35cc" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44289 } }
I20260923 17:21:44.410311 32112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:44.410459 32112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:44.410652 32112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:44.410941 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:44.385731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43114 (local address 127.28.72.254:44289)
0923 17:21:44.386654 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:44.386671 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:44.386732 (+ 61us) server_negotiation.cc:407] Connection header received
0923 17:21:44.389176 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:44.389209 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:44.389806 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:44.390301 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:44.391807 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:44.393756 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:44.395377 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:44.396222 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:44.396441 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:44.401171 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:44.401183 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:44.401212 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:44.401263 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:44.405633 (+ 4370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:44.406498 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:44.406509 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:44.406521 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:44.406768 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:44.407290 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:44.407296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:44.407300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:44.408107 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:44.408435 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:44.408866 (+ 431us) server_negotiation.cc:299] Negotiation successful
0923 17:21:44.409458 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":386,"threads_started":1}
I20260923 17:21:44.413673 32112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790e9dabcff84cdc9bb80d8a265d35cc" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44289 } }
I20260923 17:21:44.414314 32112 leader_election.cc:302] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 790e9dabcff84cdc9bb80d8a265d35cc; no voters:
I20260923 17:21:44.414913 32112 leader_election.cc:288] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:44.414999 32117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:44.415845 32117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [term 1 LEADER]: Becoming Leader. State: Replica: 790e9dabcff84cdc9bb80d8a265d35cc, State: Running, Role: LEADER
I20260923 17:21:44.416369 32117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790e9dabcff84cdc9bb80d8a265d35cc" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44289 } }
I20260923 17:21:44.417096 32112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:44.419565 32119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 790e9dabcff84cdc9bb80d8a265d35cc. Latest consensus state: current_term: 1 leader_uuid: "790e9dabcff84cdc9bb80d8a265d35cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790e9dabcff84cdc9bb80d8a265d35cc" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44289 } } }
I20260923 17:21:44.420104 32119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:44.420986 32118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "790e9dabcff84cdc9bb80d8a265d35cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790e9dabcff84cdc9bb80d8a265d35cc" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44289 } } }
I20260923 17:21:44.421304 32118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:44.423008 32126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:44.430495 32126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:44.440450 32126 catalog_manager.cc:1347] Generated new cluster ID: 0cc875eeee5247f386511e826a925adf
I20260923 17:21:44.440593 32126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:44.453612 32126 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:44.455303 32126 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:44.466102 32126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 790e9dabcff84cdc9bb80d8a265d35cc: Generated new TSK 0
I20260923 17:21:44.467268 32126 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:21:44.524804 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:44289
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:44.876015 32138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:44.876330 32138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:44.876497 32138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:44.884159 32138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:44.884284 32138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:44.884470 32138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:21:44.895004 32138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:44289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:44.896562 32138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:44.898149 32138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:44.913275 32143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:44.913462 32146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:44.913254 32144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:44.914608 32138 server_base.cc:1034] running on GCE node
I20260923 17:21:44.915405 32138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:44.916587 32138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:44.917866 32138 hybrid_clock.cc:648] HybridClock initialized: now 1790184104917792 us; error 69 us; skew 500 ppm
I20260923 17:21:44.918296 32138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:44 dist-test-slave-j0m4 krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184104, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:44.921919 32138 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:21:44.923657 32138 webserver.cc:460] Webserver started at http://127.28.72.193:41741/ using document root <none> and password file <none>
I20260923 17:21:44.924446 32138 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:44.924556 32138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:44.924831 32138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:44.927970 32138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "a4ff502f23cd4fa294ee46521efe1200"
format_stamp: "Formatted at 2026-09-23 17:21:44 on dist-test-slave-j0m4"
server_key: "bd8298ea552501df6162ef456ad4681d"
server_key_iv: "a9a9f9b3968af08718e7690aaf84be07"
server_key_version: "encryptionkey@0"
I20260923 17:21:44.928864 32138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "a4ff502f23cd4fa294ee46521efe1200"
format_stamp: "Formatted at 2026-09-23 17:21:44 on dist-test-slave-j0m4"
server_key: "bd8298ea552501df6162ef456ad4681d"
server_key_iv: "a9a9f9b3968af08718e7690aaf84be07"
server_key_version: "encryptionkey@0"
I20260923 17:21:44.934624 32138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:21:44.938747 32153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:44.940513 32138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:21:44.940717 32138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "a4ff502f23cd4fa294ee46521efe1200"
format_stamp: "Formatted at 2026-09-23 17:21:44 on dist-test-slave-j0m4"
server_key: "bd8298ea552501df6162ef456ad4681d"
server_key_iv: "a9a9f9b3968af08718e7690aaf84be07"
server_key_version: "encryptionkey@0"
I20260923 17:21:44.940968 32138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:44.961680 32138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:44.967447 32138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:44.967782 32138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:44.968899 32138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:44.970366 32138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:44.970472 32138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:44.970626 32138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:44.970700 32138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:45.007987 32138 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:34403
I20260923 17:21:45.008042 32265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:34403 every 8 connection(s)
I20260923 17:21:45.009361 32138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:21:45.017128 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 32138
I20260923 17:21:45.017364 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:21:45.018160 28963 external_mini_cluster.cc:1428] Setting key 97a8b2c07f0f2bf54b48c56f40fe4237
Sep 23 17:21:45 dist-test-slave-j0m4 krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184104, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:45.036082 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:45.012294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:60397 (local address 127.28.72.254:44289)
0923 17:21:45.012782 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:45.012786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:45.013968 (+ 1182us) server_negotiation.cc:407] Connection header received
0923 17:21:45.015969 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:45.015975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:45.016080 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:45.016208 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:45.018619 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:45.019575 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:45.022210 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:45.022701 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:45.022907 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:45.028123 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:45.028133 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:45.028141 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:45.028193 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:45.032013 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:45.032975 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:45.032985 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:45.032991 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:45.033153 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:45.033672 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:45.033679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:45.033684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:45.034132 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:45.034307 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:45.035220 (+ 913us) server_negotiation.cc:299] Negotiation successful
0923 17:21:45.035516 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":283,"threads_started":1}
I20260923 17:21:45.037009 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:45.012872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:44289 (local address 127.28.72.193:60397)
0923 17:21:45.013770 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:45.013821 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:45.015655 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:45.016494 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:45.016503 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:45.017408 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:45.018372 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:45.018407 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:45.019871 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:45.019878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:45.022008 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:45.022031 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:45.023005 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:45.023014 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:45.023441 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:45.024460 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:45.024503 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:45.027813 (+ 3310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:45.032229 (+ 4416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:45.032238 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:45.032254 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:45.032816 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:45.033303 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:45.033307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:45.033309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:45.033536 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:45.034294 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:45.034301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:45.034756 (+ 455us) client_negotiation.cc:769] Sending connection context
0923 17:21:45.035025 (+ 269us) client_negotiation.cc:240] Negotiation successful
0923 17:21:45.035675 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":401,"threads_started":1}
I20260923 17:21:45.039057 32266 heartbeater.cc:347] Connected to a master server at 127.28.72.254:44289
I20260923 17:21:45.039435 32266 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
I20260923 17:21:45.040719 32266 heartbeater.cc:511] Master 127.28.72.254:44289 requested a full tablet report, sending...
I20260923 17:21:45.043085 32077 ts_manager.cc:194] Registered new tserver with Master: a4ff502f23cd4fa294ee46521efe1200 (127.28.72.193:34403)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
I20260923 17:21:45.045440 32077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:60397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:21:45.073035 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:44289
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:45.422204 32272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:45.422518 32272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:45.422617 32272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:45.430627 32272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:45.430755 32272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:45.430871 32272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:21:45.441768 32272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:44289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:45.443104 32272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:45.444499 32272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:45.458503 32277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:45.458547 32278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:45.458603 32280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:45.459779 32272 server_base.cc:1034] running on GCE node
I20260923 17:21:45.460525 32272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:45.461627 32272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:45.462904 32272 hybrid_clock.cc:648] HybridClock initialized: now 1790184105462841 us; error 55 us; skew 500 ppm
I20260923 17:21:45.463346 32272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:45 dist-test-slave-j0m4 krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:45.467322 32272 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:21:45.469128 32272 webserver.cc:460] Webserver started at http://127.28.72.194:41559/ using document root <none> and password file <none>
I20260923 17:21:45.470001 32272 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:45.470161 32272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:45.470506 32272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:45.473798 32272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "fcede7332ecd438890d37664a86a27a9"
format_stamp: "Formatted at 2026-09-23 17:21:45 on dist-test-slave-j0m4"
server_key: "fec6f390751aa80cee89cee7ee5996a2"
server_key_iv: "b32a68738f25f04452ed933ed85c70a6"
server_key_version: "encryptionkey@0"
I20260923 17:21:45.474704 32272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "fcede7332ecd438890d37664a86a27a9"
format_stamp: "Formatted at 2026-09-23 17:21:45 on dist-test-slave-j0m4"
server_key: "fec6f390751aa80cee89cee7ee5996a2"
server_key_iv: "b32a68738f25f04452ed933ed85c70a6"
server_key_version: "encryptionkey@0"
I20260923 17:21:45.480765 32272 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 17:21:45.484870 32287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:45.486662 32272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:21:45.486937 32272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "fcede7332ecd438890d37664a86a27a9"
format_stamp: "Formatted at 2026-09-23 17:21:45 on dist-test-slave-j0m4"
server_key: "fec6f390751aa80cee89cee7ee5996a2"
server_key_iv: "b32a68738f25f04452ed933ed85c70a6"
server_key_version: "encryptionkey@0"
I20260923 17:21:45.487208 32272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:45.505117 32272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:45.509732 32272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:45.510059 32272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:45.511155 32272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:45.512594 32272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:45.512708 32272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:45.512882 32272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:45.512959 32272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:45.548163 32272 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:38669
I20260923 17:21:45.548210 32399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:38669 every 8 connection(s)
I20260923 17:21:45.549505 32272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:21:45.557657 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 32272
I20260923 17:21:45.557904 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:21:45.558434 28963 external_mini_cluster.cc:1428] Setting key d4ecd9ba5f308226c4a3e4cdc473bc88
Sep 23 17:21:45 dist-test-slave-j0m4 krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:45.572664 32402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:45.552269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:56737 (local address 127.28.72.254:44289)
0923 17:21:45.552725 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:45.552732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:45.553791 (+ 1059us) server_negotiation.cc:407] Connection header received
0923 17:21:45.555674 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:45.555679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:45.555748 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:45.555849 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:45.557505 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:45.558680 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:45.560821 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:45.561578 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:45.561740 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:45.565850 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:45.565860 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:45.565869 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:45.565917 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:45.568984 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:45.570083 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:45.570088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:45.570090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:45.570233 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:45.570764 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:45.570768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:45.570770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:45.571038 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:45.571231 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:45.572146 (+ 915us) server_negotiation.cc:299] Negotiation successful
0923 17:21:45.572325 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":224,"threads_started":1}
I20260923 17:21:45.574251 32403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:45.552896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:44289 (local address 127.28.72.194:56737)
0923 17:21:45.553627 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:45.553670 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:45.555385 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:45.556114 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:45.556123 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:45.556698 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:45.557303 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:45.557322 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:45.558862 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:45.558867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:45.560655 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:45.560670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:45.561828 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:45.561835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:45.562055 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:45.562937 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:45.562967 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:45.565569 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:45.569170 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:45.569183 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:45.569207 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:45.569922 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:45.570351 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:45.570359 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:45.570366 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:45.570629 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:45.571211 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:45.571222 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:45.571717 (+ 495us) client_negotiation.cc:769] Sending connection context
0923 17:21:45.572042 (+ 325us) client_negotiation.cc:240] Negotiation successful
0923 17:21:45.572691 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":276,"threads_started":1}
I20260923 17:21:45.575942 32400 heartbeater.cc:347] Connected to a master server at 127.28.72.254:44289
I20260923 17:21:45.576270 32400 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:45.577097 32400 heartbeater.cc:511] Master 127.28.72.254:44289 requested a full tablet report, sending...
I20260923 17:21:45.578634 32077 ts_manager.cc:194] Registered new tserver with Master: fcede7332ecd438890d37664a86a27a9 (127.28.72.194:38669)
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
I20260923 17:21:45.579577 32077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:56737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:21:45.606523 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:44289
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:45.950784 32406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:45.951201 32406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:45.951364 32406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:45.959000 32406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:45.959216 32406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:45.959393 32406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:21:45.969566 32406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:44289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:45.970996 32406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:45.972405 32406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:45.986936 32411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:45.986992 32414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:45.987092 32412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:45.987726 32406 server_base.cc:1034] running on GCE node
I20260923 17:21:45.988448 32406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:45.989616 32406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:45.990869 32406 hybrid_clock.cc:648] HybridClock initialized: now 1790184105990833 us; error 66 us; skew 500 ppm
I20260923 17:21:45.991336 32406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:45 dist-test-slave-j0m4 krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:45.994964 32406 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:21:45.996588 32406 webserver.cc:460] Webserver started at http://127.28.72.195:35715/ using document root <none> and password file <none>
I20260923 17:21:45.997344 32406 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:45.997506 32406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:45.997833 32406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:46.001104 32406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "aef831a4a697463489ce10e426d50a80"
format_stamp: "Formatted at 2026-09-23 17:21:45 on dist-test-slave-j0m4"
server_key: "f3a3a02cd191542a6312b8cbaa883933"
server_key_iv: "82a2cf353f5810a5167b9f9eaaabdb48"
server_key_version: "encryptionkey@0"
I20260923 17:21:46.001968 32406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "aef831a4a697463489ce10e426d50a80"
format_stamp: "Formatted at 2026-09-23 17:21:45 on dist-test-slave-j0m4"
server_key: "f3a3a02cd191542a6312b8cbaa883933"
server_key_iv: "82a2cf353f5810a5167b9f9eaaabdb48"
server_key_version: "encryptionkey@0"
I20260923 17:21:46.008082 32406 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260923 17:21:46.012951 32421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:46.014465 32406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:21:46.014613 32406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "aef831a4a697463489ce10e426d50a80"
format_stamp: "Formatted at 2026-09-23 17:21:45 on dist-test-slave-j0m4"
server_key: "f3a3a02cd191542a6312b8cbaa883933"
server_key_iv: "82a2cf353f5810a5167b9f9eaaabdb48"
server_key_version: "encryptionkey@0"
I20260923 17:21:46.014803 32406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:46.037175 32406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:46.041667 32406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:46.041940 32406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:46.042953 32406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:46.044380 32406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:46.044508 32406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:46.044665 32406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:46.044739 32406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:46.049496 32266 heartbeater.cc:503] Master 127.28.72.254:44289 was elected leader, sending a full tablet report...
I20260923 17:21:46.080703 32406 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:33147
I20260923 17:21:46.080755 32533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:33147 every 8 connection(s)
I20260923 17:21:46.082046 32406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:21:46.087414 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 32406
I20260923 17:21:46.087636 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:21:46.088116 28963 external_mini_cluster.cc:1428] Setting key d9898a06fbbb7e00493892e180a21319
Sep 23 17:21:46 dist-test-slave-j0m4 krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:46.103731 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:46.084823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:35735 (local address 127.28.72.254:44289)
0923 17:21:46.085268 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:46.085272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:46.086132 (+ 860us) server_negotiation.cc:407] Connection header received
0923 17:21:46.088168 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:46.088174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:46.088280 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:46.088403 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:46.090229 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:46.091306 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:46.092932 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:46.093319 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:46.093425 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:46.097201 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:46.097207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:46.097211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:46.097244 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:46.100127 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:46.100955 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:46.100965 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:46.100967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:46.101085 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:46.101551 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:46.101556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:46.101560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:46.101900 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:46.102056 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:46.103026 (+ 970us) server_negotiation.cc:299] Negotiation successful
0923 17:21:46.103283 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":236,"threads_started":1}
I20260923 17:21:46.104889 32537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:46.085257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:44289 (local address 127.28.72.195:35735)
0923 17:21:46.085970 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:46.086008 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:46.087896 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:46.088704 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:46.088717 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:46.089303 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:46.090010 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:46.090036 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:46.091479 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:46.091483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:46.092791 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:46.092803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:46.093511 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:46.093516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:46.093729 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:46.094425 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:46.094448 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:46.096964 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:46.100288 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:46.100298 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:46.100317 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:46.100764 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:46.101199 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:46.101203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:46.101207 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:46.101405 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:46.102086 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:46.102095 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:46.102595 (+ 500us) client_negotiation.cc:769] Sending connection context
0923 17:21:46.102924 (+ 329us) client_negotiation.cc:240] Negotiation successful
0923 17:21:46.103551 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":273,"threads_started":1}
I20260923 17:21:46.106287 32534 heartbeater.cc:347] Connected to a master server at 127.28.72.254:44289
I20260923 17:21:46.106630 32534 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:46.107450 32534 heartbeater.cc:511] Master 127.28.72.254:44289 requested a full tablet report, sending...
I20260923 17:21:46.109066 32077 ts_manager.cc:194] Registered new tserver with Master: aef831a4a697463489ce10e426d50a80 (127.28.72.195:33147)
I20260923 17:21:46.109936 32077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:35735
I20260923 17:21:46.117202 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:21:46 dist-test-slave-j0m4 krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184106, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 17:21:46 dist-test-slave-j0m4 krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184106, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:46.170492 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:46.154871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.28.72.254:44289)
0923 17:21:46.155147 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:46.155157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:46.155680 (+ 523us) server_negotiation.cc:407] Connection header received
0923 17:21:46.155908 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:46.155916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:46.156010 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:46.156103 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:46.157828 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:46.158679 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:46.160118 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:46.160504 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:46.160606 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:46.164865 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:46.164870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:46.164874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:46.164918 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:46.167752 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:46.168463 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:46.168468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:46.168471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:46.168595 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:46.168998 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:46.169008 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:46.169010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:46.169343 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:46.169609 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:46.169927 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 17:21:46.170172 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260923 17:21:46.171224 32077 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51102
I20260923 17:21:46.172580 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 32138
Sep 23 17:21:46 dist-test-slave-j0m4 krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184106, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:21:46.189540 32546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:46.174359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.28.72.193:34403)
0923 17:21:46.174908 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:46.174915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:46.174939 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:21:46.174994 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:46.175001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:46.175294 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:46.175442 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:46.176624 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:46.177612 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:46.178934 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:46.179580 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:46.179679 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:46.183757 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:46.183769 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:46.183784 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:46.183819 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:46.186552 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:46.187378 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:46.187386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:46.187388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:46.187535 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:46.187895 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:46.187899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:46.187901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:46.188285 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:46.188514 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:46.188852 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 17:21:46.189165 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":268,"threads_started":1}
W20260923 17:21:46.191099 32261 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49284
W20260923 17:21:46.192440 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 32138: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:21:46.192667 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 32138
I20260923 17:21:46.208698 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 32272
Sep 23 17:21:46 dist-test-slave-j0m4 krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184106, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:21:46.227013 32547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:46.209780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57856 (local address 127.28.72.194:38669)
0923 17:21:46.210385 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:46.210393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:46.210418 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:21:46.210481 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:46.210488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:46.210695 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:46.210836 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:46.212260 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:46.213570 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:46.214943 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:46.215624 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:46.215743 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:46.220450 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:46.220466 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:46.220481 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:46.220522 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:46.224032 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:46.224811 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:46.224834 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:46.224838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:46.224976 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:46.225386 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:46.225389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:46.225392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:46.225767 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:46.226015 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:46.226301 (+ 286us) server_negotiation.cc:299] Negotiation successful
0923 17:21:46.226654 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":310,"threads_started":1}
W20260923 17:21:46.228528 32395 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57856
W20260923 17:21:46.229638 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 32272: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:21:46.229789 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 32272
I20260923 17:21:46.245558 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 32406
Sep 23 17:21:46 dist-test-slave-j0m4 krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184106, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:21:46.263410 32548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:46.246617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51188 (local address 127.28.72.195:33147)
0923 17:21:46.247273 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:46.247281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:46.247309 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:21:46.247378 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:46.247385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:46.247600 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:46.247776 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:46.249168 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:46.250183 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:46.251574 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:46.252148 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:46.252288 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:46.257060 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:46.257076 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:46.257093 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:46.257140 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:46.260495 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:46.261203 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:46.261211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:46.261213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:46.261342 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:46.261853 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:46.261857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:46.261859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:46.262228 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:46.262535 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:46.262797 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 17:21:46.263019 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":312,"threads_started":1}
W20260923 17:21:46.264822 32529 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51188
W20260923 17:21:46.266099 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 32406: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:21:46.266237 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 32406
I20260923 17:21:46.281852 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 32045
I20260923 17:21:46.283238 32107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:46.386353 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 32045
2026-09-23T17:21:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3783 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:21:46 dist-test-slave-j0m4 krb5kdc[32552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:21:46 dist-test-slave-j0m4 krb5kdc[32552](info): set up 2 sockets
Sep 23 17:21:46 dist-test-slave-j0m4 krb5kdc[32552](info): commencing operation
krb5kdc: starting...
W20260923 17:21:47.507697 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.012s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:21:47 dist-test-slave-j0m4 krb5kdc[32552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:21:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:21:47.719453 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:43785
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:43785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:48.053900 32565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:48.054234 32565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:48.054421 32565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:48.061832 32565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:48.061962 32565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:48.062008 32565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:48.062125 32565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:48.062186 32565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:21:48.072428 32565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:43785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:43785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:48.073871 32565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:48.075258 32565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:48.088133 32570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:48.088160 32571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:48.088222 32573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:48.089737 32565 server_base.cc:1034] running on GCE node
I20260923 17:21:48.090781 32565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:48.092561 32565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:48.093866 32565 hybrid_clock.cc:648] HybridClock initialized: now 1790184108093817 us; error 73 us; skew 500 ppm
I20260923 17:21:48.094300 32565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:48 dist-test-slave-j0m4 krb5kdc[32552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:48.105969 32565 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:21:48.107929 32565 webserver.cc:460] Webserver started at http://127.28.72.254:36249/ using document root <none> and password file <none>
I20260923 17:21:48.108708 32565 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:48.108866 32565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:48.109215 32565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:48.112394 32565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "fd18f5f5ea8b48b3bb3a62dbe1fd534c"
format_stamp: "Formatted at 2026-09-23 17:21:48 on dist-test-slave-j0m4"
server_key: "c58bb1ddab445e4efd09675c83908773"
server_key_iv: "eeeb70eabafe6f16d67d0569ba4aa9e6"
server_key_version: "encryptionkey@0"
I20260923 17:21:48.113260 32565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "fd18f5f5ea8b48b3bb3a62dbe1fd534c"
format_stamp: "Formatted at 2026-09-23 17:21:48 on dist-test-slave-j0m4"
server_key: "c58bb1ddab445e4efd09675c83908773"
server_key_iv: "eeeb70eabafe6f16d67d0569ba4aa9e6"
server_key_version: "encryptionkey@0"
I20260923 17:21:48.118939 32565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 17:21:48.122711 32580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:48.124238 32565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:21:48.124444 32565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "fd18f5f5ea8b48b3bb3a62dbe1fd534c"
format_stamp: "Formatted at 2026-09-23 17:21:48 on dist-test-slave-j0m4"
server_key: "c58bb1ddab445e4efd09675c83908773"
server_key_iv: "eeeb70eabafe6f16d67d0569ba4aa9e6"
server_key_version: "encryptionkey@0"
I20260923 17:21:48.124652 32565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:48.183513 32565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:48.187901 32565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:48.188246 32565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:48.207409 32565 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:43785
I20260923 17:21:48.207409 32631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:43785 every 8 connection(s)
I20260923 17:21:48.208817 32565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:48.210542 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 32565
I20260923 17:21:48.210755 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:21:48.211588 28963 external_mini_cluster.cc:1428] Setting key efa19bf7816e7464d7234d76a9baad59
I20260923 17:21:48.213560 32632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:48.225785 32632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c: Bootstrap starting.
Sep 23 17:21:48 dist-test-slave-j0m4 krb5kdc[32552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:48.229539 32632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:48.230769 32632 log.cc:824] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:48.233901 32632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c: No bootstrap required, opened a new log
I20260923 17:21:48.236289 32635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:48.213934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.28.72.254:43785)
0923 17:21:48.214822 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:48.214836 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:48.214897 (+ 61us) server_negotiation.cc:407] Connection header received
0923 17:21:48.216745 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:48.216770 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:48.217242 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:48.217712 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:48.219476 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:48.220952 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:48.222369 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:48.223269 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:48.223530 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:48.228169 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:48.228178 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:48.228194 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:48.228239 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:48.231891 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:48.232662 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:48.232669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:48.232676 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:48.232869 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:48.233272 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:48.233278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:48.233281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:48.233809 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:48.234200 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:48.234730 (+ 530us) server_negotiation.cc:299] Negotiation successful
0923 17:21:48.235227 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":431,"threads_started":1}
I20260923 17:21:48.242635 32632 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd18f5f5ea8b48b3bb3a62dbe1fd534c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43785 } }
I20260923 17:21:48.243086 32632 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:48.243287 32632 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd18f5f5ea8b48b3bb3a62dbe1fd534c, State: Initialized, Role: FOLLOWER
I20260923 17:21:48.243884 32632 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd18f5f5ea8b48b3bb3a62dbe1fd534c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43785 } }
I20260923 17:21:48.244138 32632 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:48.244293 32632 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:48.244477 32632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:48.246932 32632 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd18f5f5ea8b48b3bb3a62dbe1fd534c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43785 } }
I20260923 17:21:48.247550 32632 leader_election.cc:302] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd18f5f5ea8b48b3bb3a62dbe1fd534c; no voters:
I20260923 17:21:48.248236 32632 leader_election.cc:288] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:48.248405 32637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:48.249284 32637 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [term 1 LEADER]: Becoming Leader. State: Replica: fd18f5f5ea8b48b3bb3a62dbe1fd534c, State: Running, Role: LEADER
I20260923 17:21:48.250052 32637 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd18f5f5ea8b48b3bb3a62dbe1fd534c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43785 } }
I20260923 17:21:48.250736 32632 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:48.253481 32639 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd18f5f5ea8b48b3bb3a62dbe1fd534c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd18f5f5ea8b48b3bb3a62dbe1fd534c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43785 } } }
I20260923 17:21:48.253458 32638 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd18f5f5ea8b48b3bb3a62dbe1fd534c. Latest consensus state: current_term: 1 leader_uuid: "fd18f5f5ea8b48b3bb3a62dbe1fd534c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd18f5f5ea8b48b3bb3a62dbe1fd534c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43785 } } }
I20260923 17:21:48.253921 32639 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:48.253932 32638 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:48.260124 32646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:48.265141 32646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:48.274502 32646 catalog_manager.cc:1347] Generated new cluster ID: 96d1b2c3e06a421b960266a684ced283
I20260923 17:21:48.274709 32646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:48.292906 32646 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:48.294447 32646 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:48.307363 32646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd18f5f5ea8b48b3bb3a62dbe1fd534c: Generated new TSK 0
I20260923 17:21:48.308426 32646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:21:48.362586 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:43785
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:48.702721 32658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:48.703044 32658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:48.703205 32658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:48.710640 32658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:48.710758 32658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:48.710937 32658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:21:48.721019 32658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:43785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:48.722385 32658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:48.723805 32658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:48.737484 32666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:48.737612 32663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:48.737515 32664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:48.738497 32658 server_base.cc:1034] running on GCE node
I20260923 17:21:48.739306 32658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:48.740412 32658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:48.741648 32658 hybrid_clock.cc:648] HybridClock initialized: now 1790184108741598 us; error 40 us; skew 500 ppm
I20260923 17:21:48.742018 32658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:48 dist-test-slave-j0m4 krb5kdc[32552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:48.745359 32658 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:21:48.746901 32658 webserver.cc:460] Webserver started at http://127.28.72.193:38279/ using document root <none> and password file <none>
I20260923 17:21:48.747695 32658 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:48.747799 32658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:48.748075 32658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:48.751332 32658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "39aaa002ce9a462f9f8cd4194d24e33b"
format_stamp: "Formatted at 2026-09-23 17:21:48 on dist-test-slave-j0m4"
server_key: "ee5c0ab4d7d1e7b9a76d4ee411a48e89"
server_key_iv: "40ebe0a40c009140f0541489ff5707a3"
server_key_version: "encryptionkey@0"
I20260923 17:21:48.752187 32658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "39aaa002ce9a462f9f8cd4194d24e33b"
format_stamp: "Formatted at 2026-09-23 17:21:48 on dist-test-slave-j0m4"
server_key: "ee5c0ab4d7d1e7b9a76d4ee411a48e89"
server_key_iv: "40ebe0a40c009140f0541489ff5707a3"
server_key_version: "encryptionkey@0"
I20260923 17:21:48.757712 32658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:21:48.761444 32673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:48.763252 32658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:21:48.763469 32658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "39aaa002ce9a462f9f8cd4194d24e33b"
format_stamp: "Formatted at 2026-09-23 17:21:48 on dist-test-slave-j0m4"
server_key: "ee5c0ab4d7d1e7b9a76d4ee411a48e89"
server_key_iv: "40ebe0a40c009140f0541489ff5707a3"
server_key_version: "encryptionkey@0"
I20260923 17:21:48.763710 32658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:48.791070 32658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:48.796116 32658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:48.796442 32658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:48.797549 32658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:48.799031 32658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:48.799160 32658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:48.799312 32658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:48.799386 32658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:48.834854 32658 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:39443
I20260923 17:21:48.834914 317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:39443 every 8 connection(s)
I20260923 17:21:48.836210 32658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:21:48.843256 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 32658
I20260923 17:21:48.843518 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:21:48.844172 28963 external_mini_cluster.cc:1428] Setting key c476209efdfbcd938d4764ce3b8ea4a3
Sep 23 17:21:48 dist-test-slave-j0m4 krb5kdc[32552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:48.858026 320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:48.838914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:38489 (local address 127.28.72.254:43785)
0923 17:21:48.839389 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:48.839394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:48.840499 (+ 1105us) server_negotiation.cc:407] Connection header received
0923 17:21:48.842328 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:48.842333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:48.842426 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:48.842549 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:48.844288 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:48.845071 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:48.846667 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:48.847081 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:48.847264 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:48.851288 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:48.851293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:48.851297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:48.851336 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:48.854236 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:48.855386 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:48.855391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:48.855393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:48.855603 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:48.856140 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:48.856144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:48.856146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:48.856477 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:48.856596 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:48.857609 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0923 17:21:48.857777 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":272,"threads_started":1}
I20260923 17:21:48.859699 321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:48.839605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:43785 (local address 127.28.72.193:38489)
0923 17:21:48.840294 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:48.840337 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:48.842056 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:48.842809 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:48.842818 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:48.843471 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:48.844112 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:48.844130 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:48.845272 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:48.845276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:48.846521 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:48.846533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:48.847350 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:48.847358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:48.847607 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:48.848318 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:48.848342 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:48.850944 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:48.854498 (+ 3554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:48.854513 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:48.854535 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:48.855259 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:48.855733 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:48.855741 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:48.855747 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:48.856014 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:48.856612 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:48.856623 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:48.857140 (+ 517us) client_negotiation.cc:769] Sending connection context
0923 17:21:48.857453 (+ 313us) client_negotiation.cc:240] Negotiation successful
0923 17:21:48.858063 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":269,"threads_started":1}
I20260923 17:21:48.861474 318 heartbeater.cc:347] Connected to a master server at 127.28.72.254:43785
I20260923 17:21:48.861835 318 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:48.862851 318 heartbeater.cc:511] Master 127.28.72.254:43785 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
I20260923 17:21:48.865422 32597 ts_manager.cc:194] Registered new tserver with Master: 39aaa002ce9a462f9f8cd4194d24e33b (127.28.72.193:39443)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
I20260923 17:21:48.867051 32597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:38489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:21:48.893015 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:43785
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:49.228534 324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:49.228848 324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:49.228995 324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:49.236581 324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:49.236737 324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:49.236944 324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:21:49.247272 324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:43785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:49.248687 324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:49.250075 324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:49.264462 330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:49.264433 332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:49.264407 329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:49.265451 324 server_base.cc:1034] running on GCE node
I20260923 17:21:49.266104 324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:49.267176 324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:49.268452 324 hybrid_clock.cc:648] HybridClock initialized: now 1790184109268365 us; error 57 us; skew 500 ppm
I20260923 17:21:49.268795 324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:49 dist-test-slave-j0m4 krb5kdc[32552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184109, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:49.272089 324 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:21:49.273712 324 webserver.cc:460] Webserver started at http://127.28.72.194:46129/ using document root <none> and password file <none>
I20260923 17:21:49.274439 324 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:49.274585 324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:49.274899 324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:49.277910 324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "3abe37203197467dbcc604bb5cbf70d5"
format_stamp: "Formatted at 2026-09-23 17:21:49 on dist-test-slave-j0m4"
server_key: "05958b2bca2c270ae5c0a870da1ce435"
server_key_iv: "970cc32759c3eecf49b2f9f7c0a2e136"
server_key_version: "encryptionkey@0"
I20260923 17:21:49.278766 324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "3abe37203197467dbcc604bb5cbf70d5"
format_stamp: "Formatted at 2026-09-23 17:21:49 on dist-test-slave-j0m4"
server_key: "05958b2bca2c270ae5c0a870da1ce435"
server_key_iv: "970cc32759c3eecf49b2f9f7c0a2e136"
server_key_version: "encryptionkey@0"
I20260923 17:21:49.284431 324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:21:49.288350 339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:49.289849 324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:21:49.290047 324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "3abe37203197467dbcc604bb5cbf70d5"
format_stamp: "Formatted at 2026-09-23 17:21:49 on dist-test-slave-j0m4"
server_key: "05958b2bca2c270ae5c0a870da1ce435"
server_key_iv: "970cc32759c3eecf49b2f9f7c0a2e136"
server_key_version: "encryptionkey@0"
I20260923 17:21:49.290266 324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:49.308071 324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:49.312593 324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:49.312898 324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:49.313889 324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:49.315438 324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:49.315542 324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:49.315696 324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:49.315770 324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:49.349351 324 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:34041
I20260923 17:21:49.349397 451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:34041 every 8 connection(s)
I20260923 17:21:49.350781 324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:21:49.353917 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 324
I20260923 17:21:49.354086 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:21:49.354553 28963 external_mini_cluster.cc:1428] Setting key 2fbfa101e0060d20cfea825af036ce1f
Sep 23 17:21:49 dist-test-slave-j0m4 krb5kdc[32552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184109, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:49.375687 320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:49.353815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:57077 (local address 127.28.72.254:43785)
0923 17:21:49.354050 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:49.354058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:49.355608 (+ 1550us) server_negotiation.cc:407] Connection header received
0923 17:21:49.358030 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:49.358034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:49.358103 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:49.358197 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:49.360639 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:49.361389 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:49.363858 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:49.364330 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:49.364474 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:49.369184 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:49.369190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:49.369193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:49.369229 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:49.372078 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:49.373134 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:49.373139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:49.373141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:49.373267 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:49.373823 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:49.373826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:49.373828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:49.374086 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:49.374194 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:49.375251 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0923 17:21:49.375435 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
I20260923 17:21:49.377182 454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:49.354409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:43785 (local address 127.28.72.194:57077)
0923 17:21:49.355429 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:49.355471 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:49.357672 (+ 2201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:49.358534 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:49.358549 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:49.359449 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:49.360410 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:49.360456 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:49.361583 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:49.361591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:49.363674 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:49.363686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:49.364584 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:49.364591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:49.364945 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:49.365775 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:49.365810 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:49.368897 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:49.372239 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:49.372249 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:49.372270 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:49.372983 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:49.373392 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:49.373401 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:49.373406 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:49.373705 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:49.374213 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:49.374222 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:49.374759 (+ 537us) client_negotiation.cc:769] Sending connection context
0923 17:21:49.375106 (+ 347us) client_negotiation.cc:240] Negotiation successful
0923 17:21:49.375744 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":340,"threads_started":1}
I20260923 17:21:49.378850 452 heartbeater.cc:347] Connected to a master server at 127.28.72.254:43785
I20260923 17:21:49.379230 452 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
I20260923 17:21:49.380074 452 heartbeater.cc:511] Master 127.28.72.254:43785 requested a full tablet report, sending...
I20260923 17:21:49.381594 32597 ts_manager.cc:194] Registered new tserver with Master: 3abe37203197467dbcc604bb5cbf70d5 (127.28.72.194:34041)
I20260923 17:21:49.382421 32597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:57077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:21:49.404423 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:43785
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:49.732353 457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:49.732725 457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:49.732898 457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:49.740567 457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:49.740746 457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:49.740934 457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:21:49.751243 457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:43785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:49.752861 457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:49.754438 457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:49.768994 463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:49.769024 462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:49.769320 465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:49.769757 457 server_base.cc:1034] running on GCE node
I20260923 17:21:49.770545 457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:49.771684 457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:49.772918 457 hybrid_clock.cc:648] HybridClock initialized: now 1790184109772850 us; error 57 us; skew 500 ppm
I20260923 17:21:49.773226 457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:49 dist-test-slave-j0m4 krb5kdc[32552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184109, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:49.781802 457 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:21:49.783607 457 webserver.cc:460] Webserver started at http://127.28.72.195:38023/ using document root <none> and password file <none>
I20260923 17:21:49.784340 457 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:49.784448 457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:49.784719 457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:49.787813 457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "86ce7ee4ed4844d9ada3cada44e0a6ab"
format_stamp: "Formatted at 2026-09-23 17:21:49 on dist-test-slave-j0m4"
server_key: "eb637be171564ebe5ba3461973c58530"
server_key_iv: "ea19542ed3ec75500807a4a118683cd0"
server_key_version: "encryptionkey@0"
I20260923 17:21:49.788578 457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "86ce7ee4ed4844d9ada3cada44e0a6ab"
format_stamp: "Formatted at 2026-09-23 17:21:49 on dist-test-slave-j0m4"
server_key: "eb637be171564ebe5ba3461973c58530"
server_key_iv: "ea19542ed3ec75500807a4a118683cd0"
server_key_version: "encryptionkey@0"
I20260923 17:21:49.794266 457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:21:49.798195 472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:49.799782 457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:21:49.800001 457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "86ce7ee4ed4844d9ada3cada44e0a6ab"
format_stamp: "Formatted at 2026-09-23 17:21:49 on dist-test-slave-j0m4"
server_key: "eb637be171564ebe5ba3461973c58530"
server_key_iv: "ea19542ed3ec75500807a4a118683cd0"
server_key_version: "encryptionkey@0"
I20260923 17:21:49.800216 457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:49.813602 457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:49.818243 457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:49.818562 457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:49.819681 457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:49.821058 457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:49.821128 457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:49.821290 457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:49.821344 457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:49.870659 318 heartbeater.cc:503] Master 127.28.72.254:43785 was elected leader, sending a full tablet report...
I20260923 17:21:49.875049 457 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:38651
I20260923 17:21:49.875061 584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:38651 every 8 connection(s)
I20260923 17:21:49.876957 457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:21:49.884425 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 457
I20260923 17:21:49.884562 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:21:49.884976 28963 external_mini_cluster.cc:1428] Setting key c14951cb5b7c649471896c3359efaf1a
Sep 23 17:21:49 dist-test-slave-j0m4 krb5kdc[32552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184109, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:49.900187 587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:49.879797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:48489 (local address 127.28.72.254:43785)
0923 17:21:49.880298 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:49.880323 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:49.881552 (+ 1229us) server_negotiation.cc:407] Connection header received
0923 17:21:49.883805 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:49.883811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:49.883902 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:49.884038 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:49.886140 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:49.887209 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:49.889484 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:49.889908 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:49.890046 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:49.894484 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:49.894491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:49.894496 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:49.894539 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:49.896937 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:49.897712 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:49.897718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:49.897722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:49.897852 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:49.898301 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:49.898308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:49.898311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:49.898586 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:49.898741 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:49.899557 (+ 816us) server_negotiation.cc:299] Negotiation successful
0923 17:21:49.899774 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":283,"threads_started":1}
I20260923 17:21:49.901230 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:49.880602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:43785 (local address 127.28.72.195:48489)
0923 17:21:49.881391 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:49.881433 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:49.883436 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:49.884308 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:49.884318 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:49.885004 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:49.885920 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:49.885942 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:49.888011 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:49.888016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:49.889314 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:49.889327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:49.890156 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:49.890164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:49.890522 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:49.891442 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:49.891473 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:49.894232 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:49.897089 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:49.897099 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:49.897118 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:49.897584 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:49.897980 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:49.897992 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:49.897996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:49.898187 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:49.898715 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:49.898723 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:49.899102 (+ 379us) client_negotiation.cc:769] Sending connection context
0923 17:21:49.899458 (+ 356us) client_negotiation.cc:240] Negotiation successful
0923 17:21:49.900017 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":330,"threads_started":1}
I20260923 17:21:49.902544 585 heartbeater.cc:347] Connected to a master server at 127.28.72.254:43785
I20260923 17:21:49.902868 585 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:49.903690 585 heartbeater.cc:511] Master 127.28.72.254:43785 requested a full tablet report, sending...
I20260923 17:21:49.905397 32597 ts_manager.cc:194] Registered new tserver with Master: 86ce7ee4ed4844d9ada3cada44e0a6ab (127.28.72.195:38651)
I20260923 17:21:49.906301 32597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:48489
I20260923 17:21:49.913118 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:21:49 dist-test-slave-j0m4 krb5kdc[32552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:21:49 dist-test-slave-j0m4 krb5kdc[32552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:21:49.967528 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:49.950142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.28.72.193:39443)
0923 17:21:49.950991 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:49.950998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:49.951025 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:21:49.951083 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:49.951090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:49.951399 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:49.951530 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:49.953098 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:49.954206 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:49.955662 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:49.956309 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:49.956493 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:49.960653 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:49.960668 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:49.960684 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:49.960726 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:49.964023 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:49.964955 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:49.964963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:49.964966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:49.965128 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:49.965678 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:49.965683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:49.965685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:49.966068 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:49.966305 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:49.966784 (+ 479us) server_negotiation.cc:299] Negotiation successful
0923 17:21:49.967019 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":423,"threads_started":1}
Sep 23 17:21:49 dist-test-slave-j0m4 krb5kdc[32552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:49.987733 587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:49.972993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38526 (local address 127.28.72.254:43785)
0923 17:21:49.973223 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:49.973237 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:49.973470 (+ 233us) server_negotiation.cc:407] Connection header received
0923 17:21:49.973671 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:49.973675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:49.973759 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:49.973875 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:49.975402 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:49.976198 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:49.977551 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:49.978034 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:49.978182 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:49.982354 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:49.982359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:49.982363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:49.982404 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:49.985124 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:49.985822 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:49.985826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:49.985829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:49.985960 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:49.986333 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:49.986337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:49.986340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:49.986732 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:49.986936 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:49.987329 (+ 393us) server_negotiation.cc:299] Negotiation successful
0923 17:21:49.987503 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260923 17:21:49.988399 32597 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38526
I20260923 17:21:49.989715 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 32658
I20260923 17:21:50.003881 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:49.990680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51634 (local address 127.28.72.193:39443)
0923 17:21:49.990927 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:49.990934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:49.991181 (+ 247us) server_negotiation.cc:407] Connection header received
0923 17:21:49.991379 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:49.991384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:49.991488 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:49.991624 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:49.992863 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:49.993713 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:49.995241 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:49.995644 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:49.995745 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:49.998179 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:49.998184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:49.998187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:49.998223 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:50.000923 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:50.001778 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:50.001782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:50.001785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:50.001934 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:50.002334 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:50.002338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:50.002340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:50.002667 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:50.002853 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:50.003290 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 17:21:50.003527 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:21:50.004664 313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:50.089182 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 32658
I20260923 17:21:50.106318 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 324
Sep 23 17:21:50 dist-test-slave-j0m4 krb5kdc[32552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:21:50.124547 598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:50.107471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.28.72.194:34041)
0923 17:21:50.108052 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:50.108061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:50.108087 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:21:50.108152 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:50.108159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:50.108372 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:50.108590 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:50.110008 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:50.111299 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:50.112718 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:50.113375 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:50.113514 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:50.118306 (+ 4792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:50.118327 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:50.118345 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:50.118390 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:50.121450 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:50.122222 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:50.122231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:50.122234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:50.122423 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:50.122894 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:50.122898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:50.122900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:50.123369 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:50.123608 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:50.123900 (+ 292us) server_negotiation.cc:299] Negotiation successful
0923 17:21:50.124170 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":275,"threads_started":1}
I20260923 17:21:50.125950 447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:50.212555 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 324
I20260923 17:21:50.229182 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 457
Sep 23 17:21:50 dist-test-slave-j0m4 krb5kdc[32552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:21:50.246460 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:50.230274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41058 (local address 127.28.72.195:38651)
0923 17:21:50.230896 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:50.230903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:50.230931 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:21:50.230986 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:50.230992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:50.231268 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:50.231408 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:50.232617 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:50.233637 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:50.234937 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:50.235739 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:50.235841 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:50.240323 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:50.240337 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:50.240354 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:50.240395 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:50.243480 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:50.244229 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:50.244237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:50.244240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:50.244368 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:50.244788 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:50.244793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:50.244797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:50.245159 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:50.245475 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:50.245810 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 17:21:50.246097 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":323,"threads_started":1}
I20260923 17:21:50.247856 580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:50.330876 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 457
I20260923 17:21:50.347239 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 32565
I20260923 17:21:50.348595 32627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:50.450843 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 32565
2026-09-23T17:21:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4063 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:21:50 dist-test-slave-j0m4 krb5kdc[605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:21:50 dist-test-slave-j0m4 krb5kdc[605](info): set up 2 sockets
Sep 23 17:21:50 dist-test-slave-j0m4 krb5kdc[605](info): commencing operation
krb5kdc: starting...
W20260923 17:21:51.563805 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:21:51 dist-test-slave-j0m4 krb5kdc[605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:21:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:21:51.783869 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:42103
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:42103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:52.119349 618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:52.119709 618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:52.119860 618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:52.127429 618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:52.127600 618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:52.127705 618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:52.127807 618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:52.127885 618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:21:52.138279 618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:42103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:42103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:52.139707 618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:52.141052 618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:52.153102 624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:52.153054 626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:52.153105 623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:52.153937 618 server_base.cc:1034] running on GCE node
I20260923 17:21:52.154779 618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:52.156323 618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:52.157569 618 hybrid_clock.cc:648] HybridClock initialized: now 1790184112157518 us; error 37 us; skew 500 ppm
I20260923 17:21:52.157944 618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:52 dist-test-slave-j0m4 krb5kdc[605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184112, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:52.161273 618 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:21:52.162531 618 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:52.162622 618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:52.162987 618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:52.166095 618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "314c38c84c264892b7f414db30a9c9f4"
format_stamp: "Formatted at 2026-09-23 17:21:52 on dist-test-slave-j0m4"
server_key: "ed1dbd56070bb3e040ab8b31fcbfd9ed"
server_key_iv: "939ef5d9318c9d35d28cc9566e8e75d4"
server_key_version: "encryptionkey@0"
I20260923 17:21:52.166927 618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "314c38c84c264892b7f414db30a9c9f4"
format_stamp: "Formatted at 2026-09-23 17:21:52 on dist-test-slave-j0m4"
server_key: "ed1dbd56070bb3e040ab8b31fcbfd9ed"
server_key_iv: "939ef5d9318c9d35d28cc9566e8e75d4"
server_key_version: "encryptionkey@0"
I20260923 17:21:52.172164 618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:21:52.175838 632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:52.177202 618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:21:52.177392 618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "314c38c84c264892b7f414db30a9c9f4"
format_stamp: "Formatted at 2026-09-23 17:21:52 on dist-test-slave-j0m4"
server_key: "ed1dbd56070bb3e040ab8b31fcbfd9ed"
server_key_iv: "939ef5d9318c9d35d28cc9566e8e75d4"
server_key_version: "encryptionkey@0"
I20260923 17:21:52.177632 618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:52.200722 618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:52.205062 618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:52.205341 618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:52.224833 618 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:42103
I20260923 17:21:52.224813 683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:42103 every 8 connection(s)
I20260923 17:21:52.225829 618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:52.229642 684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:52.234831 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 618
I20260923 17:21:52.235085 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:21:52.235767 28963 external_mini_cluster.cc:1428] Setting key c737977c2d2199ca6a81a11bd695f3c7
I20260923 17:21:52.240638 684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4: Bootstrap starting.
I20260923 17:21:52.244491 684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:52.245745 684 log.cc:824] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:52.249111 684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4: No bootstrap required, opened a new log
Sep 23 17:21:52 dist-test-slave-j0m4 krb5kdc[605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:52.257741 684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314c38c84c264892b7f414db30a9c9f4" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42103 } }
I20260923 17:21:52.258157 684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:52.258285 684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 314c38c84c264892b7f414db30a9c9f4, State: Initialized, Role: FOLLOWER
I20260923 17:21:52.259035 684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314c38c84c264892b7f414db30a9c9f4" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42103 } }
I20260923 17:21:52.259449 684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:52.259591 684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:52.259794 684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:52.260598 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:52.237875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.28.72.254:42103)
0923 17:21:52.238743 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:52.238755 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:52.238796 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:21:52.240611 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:52.240640 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:52.241103 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:52.241469 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:52.243438 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:52.244837 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:52.246659 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:52.247621 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:52.247755 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:52.252590 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:52.252598 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:52.252613 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:52.252655 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:52.255969 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:52.256704 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:52.256712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:52.256718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:52.256879 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:52.257254 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:52.257259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:52.257262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:52.257861 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:52.258159 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:52.258615 (+ 456us) server_negotiation.cc:299] Negotiation successful
0923 17:21:52.259075 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":419,"threads_started":1}
I20260923 17:21:52.262508 684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314c38c84c264892b7f414db30a9c9f4" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42103 } }
I20260923 17:21:52.263016 684 leader_election.cc:302] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 314c38c84c264892b7f414db30a9c9f4; no voters:
I20260923 17:21:52.263734 684 leader_election.cc:288] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:52.263841 689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:52.264520 689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [term 1 LEADER]: Becoming Leader. State: Replica: 314c38c84c264892b7f414db30a9c9f4, State: Running, Role: LEADER
I20260923 17:21:52.265120 689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314c38c84c264892b7f414db30a9c9f4" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42103 } }
I20260923 17:21:52.265978 684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:52.268085 691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 314c38c84c264892b7f414db30a9c9f4. Latest consensus state: current_term: 1 leader_uuid: "314c38c84c264892b7f414db30a9c9f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314c38c84c264892b7f414db30a9c9f4" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42103 } } }
I20260923 17:21:52.268527 690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "314c38c84c264892b7f414db30a9c9f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314c38c84c264892b7f414db30a9c9f4" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42103 } } }
I20260923 17:21:52.270656 690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:52.271042 691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:52.272440 699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:52.277560 699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:52.287603 699 catalog_manager.cc:1347] Generated new cluster ID: 75a1282490d34358a2c316b6942e6357
I20260923 17:21:52.287750 699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:52.295995 699 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:52.297364 699 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:52.310478 699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4: Generated new TSK 0
I20260923 17:21:52.311442 699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:21:52.371197 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42103
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:52.718334 710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:52.718676 710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:52.718866 710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:52.726347 710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:52.726518 710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:52.726714 710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:21:52.737879 710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:52.739435 710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:52.740890 710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:52.755085 715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:52.755297 716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:52.755065 718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:52.755721 710 server_base.cc:1034] running on GCE node
I20260923 17:21:52.756471 710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:52.757412 710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:52.758643 710 hybrid_clock.cc:648] HybridClock initialized: now 1790184112758566 us; error 66 us; skew 500 ppm
I20260923 17:21:52.759171 710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:52 dist-test-slave-j0m4 krb5kdc[605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184112, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:52.762593 710 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:21:52.763835 710 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:52.763940 710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:52.764204 710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:52.767426 710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "770e74f0729f4f13b4346a856ef17522"
format_stamp: "Formatted at 2026-09-23 17:21:52 on dist-test-slave-j0m4"
server_key: "d00cef0edc5569d8457ff419d0a38909"
server_key_iv: "3845e38ec08e908cb2f52959ff993942"
server_key_version: "encryptionkey@0"
I20260923 17:21:52.768251 710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "770e74f0729f4f13b4346a856ef17522"
format_stamp: "Formatted at 2026-09-23 17:21:52 on dist-test-slave-j0m4"
server_key: "d00cef0edc5569d8457ff419d0a38909"
server_key_iv: "3845e38ec08e908cb2f52959ff993942"
server_key_version: "encryptionkey@0"
I20260923 17:21:52.774194 710 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 17:21:52.777846 724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:52.779562 710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 17:21:52.779764 710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "770e74f0729f4f13b4346a856ef17522"
format_stamp: "Formatted at 2026-09-23 17:21:52 on dist-test-slave-j0m4"
server_key: "d00cef0edc5569d8457ff419d0a38909"
server_key_iv: "3845e38ec08e908cb2f52959ff993942"
server_key_version: "encryptionkey@0"
I20260923 17:21:52.780000 710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:52.819530 710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:52.824317 710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:52.824635 710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:52.825728 710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:52.827179 710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:52.827292 710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:52.827420 710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:52.827525 710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:52.862385 710 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:40727
I20260923 17:21:52.862527 836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:40727 every 8 connection(s)
I20260923 17:21:52.863340 710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:21:52.865061 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 710
I20260923 17:21:52.865258 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:21:52.865958 28963 external_mini_cluster.cc:1428] Setting key fa26c524f67f43f26f55de33fa89a323
Sep 23 17:21:52 dist-test-slave-j0m4 krb5kdc[605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184112, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:52.888684 839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:52.866024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:39571 (local address 127.28.72.254:42103)
0923 17:21:52.866542 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:52.866549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:52.867777 (+ 1228us) server_negotiation.cc:407] Connection header received
0923 17:21:52.870448 (+ 2671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:52.870452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:52.870521 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:52.870707 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:52.872924 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:52.873770 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:52.875929 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:52.876748 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:52.876925 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:52.880603 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:52.880613 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:52.880622 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:52.880674 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:52.884780 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:52.885701 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:52.885711 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:52.885718 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:52.885908 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:52.886336 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:52.886345 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:52.886350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:52.886764 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:52.886964 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:52.887911 (+ 947us) server_negotiation.cc:299] Negotiation successful
0923 17:21:52.888181 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":246,"threads_started":1}
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
I20260923 17:21:52.889463 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:52.866798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:42103 (local address 127.28.72.193:39571)
0923 17:21:52.867623 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:52.867666 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:52.870107 (+ 2441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:52.870942 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:52.870957 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:52.871801 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:52.872708 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:52.872738 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:52.873996 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:52.874012 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:52.875752 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:52.875771 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:52.877009 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:52.877014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:52.877221 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:52.877981 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:52.878027 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:52.880312 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:52.884967 (+ 4655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:52.884975 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:52.884988 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:52.885547 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:52.886029 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:52.886032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:52.886034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:52.886204 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:52.886943 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:52.886950 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:52.887448 (+ 498us) client_negotiation.cc:769] Sending connection context
0923 17:21:52.887715 (+ 267us) client_negotiation.cc:240] Negotiation successful
0923 17:21:52.888155 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":370,"threads_started":1}
I20260923 17:21:52.891316 837 heartbeater.cc:347] Connected to a master server at 127.28.72.254:42103
I20260923 17:21:52.891678 837 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
I20260923 17:21:52.892611 837 heartbeater.cc:511] Master 127.28.72.254:42103 requested a full tablet report, sending...
I20260923 17:21:52.894886 649 ts_manager.cc:194] Registered new tserver with Master: 770e74f0729f4f13b4346a856ef17522 (127.28.72.193:40727)
I20260923 17:21:52.896548 649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:39571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:21:52.919065 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42103
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:53.259562 843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:53.259843 843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:53.259940 843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:53.267396 843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:53.267529 843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:53.267645 843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:21:53.278012 843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:53.279345 843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:53.280632 843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:53.294641 849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:53.294655 848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:53.294687 851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:53.295768 843 server_base.cc:1034] running on GCE node
I20260923 17:21:53.296491 843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:53.297642 843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:53.298952 843 hybrid_clock.cc:648] HybridClock initialized: now 1790184113298875 us; error 76 us; skew 500 ppm
I20260923 17:21:53.299568 843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:53 dist-test-slave-j0m4 krb5kdc[605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:53.307655 843 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:21:53.309157 843 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:53.309314 843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:53.309643 843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:53.313030 843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "1ef8166903934c0892f639ed359a0517"
format_stamp: "Formatted at 2026-09-23 17:21:53 on dist-test-slave-j0m4"
server_key: "b6fabc8e07263af0b808c41ebcea6609"
server_key_iv: "f178dfa3be28a1befd0f3573123ceb90"
server_key_version: "encryptionkey@0"
I20260923 17:21:53.313938 843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "1ef8166903934c0892f639ed359a0517"
format_stamp: "Formatted at 2026-09-23 17:21:53 on dist-test-slave-j0m4"
server_key: "b6fabc8e07263af0b808c41ebcea6609"
server_key_iv: "f178dfa3be28a1befd0f3573123ceb90"
server_key_version: "encryptionkey@0"
I20260923 17:21:53.320586 843 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 17:21:53.324385 857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:53.325891 843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:21:53.326098 843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "1ef8166903934c0892f639ed359a0517"
format_stamp: "Formatted at 2026-09-23 17:21:53 on dist-test-slave-j0m4"
server_key: "b6fabc8e07263af0b808c41ebcea6609"
server_key_iv: "f178dfa3be28a1befd0f3573123ceb90"
server_key_version: "encryptionkey@0"
I20260923 17:21:53.326342 843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:53.343055 843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:53.347968 843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:53.348285 843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:53.349324 843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:53.351225 843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:53.351361 843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 17:21:53.351502 843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:53.351591 843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:53.386835 843 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:44449
I20260923 17:21:53.386869 969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:44449 every 8 connection(s)
I20260923 17:21:53.387761 843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:21:53.390784 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 843
I20260923 17:21:53.390973 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:21:53.391480 28963 external_mini_cluster.cc:1428] Setting key 9cd096a42d0c10da9222ee3496c04c23
Sep 23 17:21:53 dist-test-slave-j0m4 krb5kdc[605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:53.412629 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:53.390513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:60391 (local address 127.28.72.254:42103)
0923 17:21:53.391059 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:53.391065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:53.392257 (+ 1192us) server_negotiation.cc:407] Connection header received
0923 17:21:53.394596 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:53.394600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:53.394674 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:53.394768 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:53.397092 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.397839 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:53.400447 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.401205 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:53.401363 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:53.406122 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:53.406127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:53.406130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:53.406165 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:53.408926 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:53.409982 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:53.409987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:53.409989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:53.410122 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:53.410645 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:53.410649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:53.410651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:53.410932 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:53.411040 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:53.412114 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0923 17:21:53.412291 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":233,"threads_started":1}
I20260923 17:21:53.414211 973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:53.391186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:42103 (local address 127.28.72.194:60391)
0923 17:21:53.392042 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:53.392101 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:53.394229 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:53.395065 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:53.395080 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:53.395965 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:53.396882 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:53.396911 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.397997 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:53.398004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:53.400267 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:53.400281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.401525 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:53.401532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:53.401739 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:53.402511 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:53.402552 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:53.405811 (+ 3259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:53.409091 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:53.409104 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:53.409127 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:53.409834 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:53.410241 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:53.410247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:53.410253 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:53.410524 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:53.411070 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:53.411082 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:53.411658 (+ 576us) client_negotiation.cc:769] Sending connection context
0923 17:21:53.412002 (+ 344us) client_negotiation.cc:240] Negotiation successful
0923 17:21:53.412611 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":315,"threads_started":1}
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
I20260923 17:21:53.415972 970 heartbeater.cc:347] Connected to a master server at 127.28.72.254:42103
I20260923 17:21:53.416260 970 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:53.417002 970 heartbeater.cc:511] Master 127.28.72.254:42103 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
I20260923 17:21:53.418649 649 ts_manager.cc:194] Registered new tserver with Master: 1ef8166903934c0892f639ed359a0517 (127.28.72.194:44449)
I20260923 17:21:53.419543 649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:60391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:21:53.443745 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42103
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:53.778532 976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:53.778824 976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:53.778996 976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:53.786514 976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:53.786639 976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:53.786870 976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:21:53.797014 976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:53.798406 976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:53.799767 976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:53.813323 981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:53.813375 984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:53.813364 982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:53.814013 976 server_base.cc:1034] running on GCE node
I20260923 17:21:53.814816 976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:53.815920 976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:53.817164 976 hybrid_clock.cc:648] HybridClock initialized: now 1790184113817108 us; error 53 us; skew 500 ppm
I20260923 17:21:53.817510 976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:21:53 dist-test-slave-j0m4 krb5kdc[605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:53.820869 976 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:21:53.822145 976 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:53.822283 976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:53.822602 976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:53.825625 976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "0cbcf73bcb4741078c6b36d1c242fdcb"
format_stamp: "Formatted at 2026-09-23 17:21:53 on dist-test-slave-j0m4"
server_key: "a3f85b6b5cc683766f313531f30f2ec1"
server_key_iv: "500b2e6ffd4cefcde8652db516d4655e"
server_key_version: "encryptionkey@0"
I20260923 17:21:53.826445 976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "0cbcf73bcb4741078c6b36d1c242fdcb"
format_stamp: "Formatted at 2026-09-23 17:21:53 on dist-test-slave-j0m4"
server_key: "a3f85b6b5cc683766f313531f30f2ec1"
server_key_iv: "500b2e6ffd4cefcde8652db516d4655e"
server_key_version: "encryptionkey@0"
I20260923 17:21:53.831871 976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:21:53.835819 990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:53.837451 976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:21:53.837682 976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "0cbcf73bcb4741078c6b36d1c242fdcb"
format_stamp: "Formatted at 2026-09-23 17:21:53 on dist-test-slave-j0m4"
server_key: "a3f85b6b5cc683766f313531f30f2ec1"
server_key_iv: "500b2e6ffd4cefcde8652db516d4655e"
server_key_version: "encryptionkey@0"
I20260923 17:21:53.837942 976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:53.862871 976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:53.867974 976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:53.868280 976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:53.869252 976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:53.870682 976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:53.870805 976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:53.870941 976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:53.871028 976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:53.900818 837 heartbeater.cc:503] Master 127.28.72.254:42103 was elected leader, sending a full tablet report...
I20260923 17:21:53.911711 976 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:36639
I20260923 17:21:53.911759 1102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:36639 every 8 connection(s)
I20260923 17:21:53.912570 976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:21:53.914317 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 976
I20260923 17:21:53.914502 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:21:53.915077 28963 external_mini_cluster.cc:1428] Setting key 89d2714176eca95c451b1f1bd92504eb
Sep 23 17:21:53 dist-test-slave-j0m4 krb5kdc[605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:21:53.934479 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:53.916360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:59793 (local address 127.28.72.254:42103)
0923 17:21:53.916804 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:53.916810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:53.916830 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:21:53.919048 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:53.919052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:53.919177 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:53.919291 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:53.920937 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.921796 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:53.923563 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.924002 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:53.924113 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:53.927832 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:53.927837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:53.927840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:53.927877 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:53.931424 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:53.932189 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:53.932192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:53.932195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:53.932311 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:53.932748 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:53.932751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:53.932753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:53.933010 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:53.933123 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:53.933979 (+ 856us) server_negotiation.cc:299] Negotiation successful
0923 17:21:53.934141 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":709,"threads_started":1}
I20260923 17:21:53.935719 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:53.915839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:42103 (local address 127.28.72.195:59793)
0923 17:21:53.916681 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:53.916739 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:53.918773 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:53.919550 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:53.919560 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:53.920143 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:53.920739 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:53.920757 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.921972 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:53.921976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:53.923417 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:53.923431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.924209 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:53.924214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:53.924437 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:53.925182 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:53.925207 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:53.927604 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:53.931572 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:53.931582 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:53.931607 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:53.932050 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:53.932428 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:53.932431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:53.932433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:53.932643 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:53.933156 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:53.933163 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:53.933611 (+ 448us) client_negotiation.cc:769] Sending connection context
0923 17:21:53.933869 (+ 258us) client_negotiation.cc:240] Negotiation successful
0923 17:21:53.934339 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":339,"threads_started":1}
I20260923 17:21:53.937204 1103 heartbeater.cc:347] Connected to a master server at 127.28.72.254:42103
I20260923 17:21:53.937609 1103 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:53.938289 1103 heartbeater.cc:511] Master 127.28.72.254:42103 requested a full tablet report, sending...
I20260923 17:21:53.939826 649 ts_manager.cc:194] Registered new tserver with Master: 0cbcf73bcb4741078c6b36d1c242fdcb (127.28.72.195:36639)
I20260923 17:21:53.940639 649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:59793
I20260923 17:21:53.944955 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:21:53.965010 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:53.950308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.28.72.254:42103)
0923 17:21:53.950687 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:53.950695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:53.951540 (+ 845us) server_negotiation.cc:407] Connection header received
0923 17:21:53.951773 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:53.951778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:53.951842 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:53.951948 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:53.953576 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.954323 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:53.955733 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.956147 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:53.956293 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:53.958714 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:53.958719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:53.958723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:53.958759 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:53.961863 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:53.962821 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:53.962828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:53.962837 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:53.962969 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:53.963416 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:53.963423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:53.963426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:53.963833 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:53.964095 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:53.964441 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 17:21:53.964675 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260923 17:21:53.970491 649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:21:53.978169 649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:21:54.003320 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:53.993981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.28.72.195:36639)
0923 17:21:53.995026 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:53.995034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:53.995066 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:21:53.995189 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:53.995198 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:53.995452 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:53.996376 (+ 924us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:53.996684 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.997977 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.000825 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.001890 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.002075 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.002358 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:54.002668 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 17:21:54.002877 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":672,"threads_started":1}
I20260923 17:21:54.003461 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:53.992382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:44449 (local address 127.0.0.1:45446)
0923 17:21:53.993273 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:53.993326 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:53.993555 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:53.994420 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:53.994432 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:53.994473 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:53.994851 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:53.994876 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.997301 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:53.997309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:54.000242 (+ 2933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:54.000255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.002281 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.002290 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:54.002517 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.002576 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 17:21:54.002650 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 17:21:54.002953 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":119,"thread_start_us":303,"threads_started":1}
I20260923 17:21:54.003597 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:53.992682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.28.72.194:44449)
0923 17:21:53.993291 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:53.993302 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:53.993456 (+ 154us) server_negotiation.cc:407] Connection header received
0923 17:21:53.993806 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:53.993818 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:53.994145 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:53.995063 (+ 918us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:53.995318 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.997141 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.000461 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.001980 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.002778 (+ 798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.002967 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:54.003035 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 17:21:54.003232 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":522,"threads_started":1}
I20260923 17:21:54.004528 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:53.993656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36639 (local address 127.0.0.1:52942)
0923 17:21:53.994214 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:53.994232 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:53.994382 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:53.995676 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:53.995683 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:53.995722 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:53.996170 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:53.996182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.998127 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:53.998135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:54.000661 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:54.000673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.002119 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.002125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:54.002396 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.002467 (+ 71us) client_negotiation.cc:769] Sending connection context
0923 17:21:54.002635 (+ 168us) client_negotiation.cc:240] Negotiation successful
0923 17:21:54.004064 (+ 1429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":1252,"thread_start_us":272,"threads_started":1}
I20260923 17:21:54.007383 1120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:53.994291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:40727 (local address 127.0.0.1:58150)
0923 17:21:53.998665 (+ 4374us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:53.998684 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:53.998819 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:53.999704 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:53.999709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:53.999733 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:54.000242 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:54.000255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.002196 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.002201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:54.004469 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:54.004488 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.006465 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.006471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:54.006599 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.006675 (+ 76us) client_negotiation.cc:769] Sending connection context
0923 17:21:54.006783 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 17:21:54.006946 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4223,"mutex_wait_us":35,"thread_start_us":4030,"threads_started":1}
I20260923 17:21:54.008061 1121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:53.994947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.28.72.193:40727)
0923 17:21:53.996948 (+ 2001us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:53.996960 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:53.998810 (+ 1850us) server_negotiation.cc:407] Connection header received
0923 17:21:53.999010 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:53.999018 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:53.999263 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:53.999456 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:54.000452 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.002005 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.004669 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.006160 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.006308 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.006515 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:54.007632 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0923 17:21:54.007816 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1828,"thread_start_us":1687,"threads_started":1}
I20260923 17:21:54.008066 905 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3a2e278002418a97498069bdc199c8 (DEFAULT_TABLE table=test-table [id=dbd2a9c6015a4e53866795fdbf582ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:54.009640 905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3a2e278002418a97498069bdc199c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:54.010399 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3a2e278002418a97498069bdc199c8 (DEFAULT_TABLE table=test-table [id=dbd2a9c6015a4e53866795fdbf582ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:54.011335 772 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3a2e278002418a97498069bdc199c8 (DEFAULT_TABLE table=test-table [id=dbd2a9c6015a4e53866795fdbf582ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:54.011855 1038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3a2e278002418a97498069bdc199c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:54.012419 772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3a2e278002418a97498069bdc199c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:54.024889 1122 tablet_bootstrap.cc:492] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522: Bootstrap starting.
I20260923 17:21:54.028326 1122 tablet_bootstrap.cc:654] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:54.029386 1122 log.cc:824] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:54.030400 1124 tablet_bootstrap.cc:492] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb: Bootstrap starting.
I20260923 17:21:54.031565 1123 tablet_bootstrap.cc:492] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517: Bootstrap starting.
I20260923 17:21:54.033900 1122 tablet_bootstrap.cc:492] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522: No bootstrap required, opened a new log
I20260923 17:21:54.034186 1122 ts_tablet_manager.cc:1397] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260923 17:21:54.034577 1124 tablet_bootstrap.cc:654] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:54.035765 1123 tablet_bootstrap.cc:654] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:54.035863 1124 log.cc:824] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:54.037000 1123 log.cc:824] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:54.038991 1124 tablet_bootstrap.cc:492] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb: No bootstrap required, opened a new log
I20260923 17:21:54.039319 1124 ts_tablet_manager.cc:1397] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 17:21:54.040025 1123 tablet_bootstrap.cc:492] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517: No bootstrap required, opened a new log
I20260923 17:21:54.040274 1123 ts_tablet_manager.cc:1397] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20260923 17:21:54.042780 1122 raft_consensus.cc:348] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8166903934c0892f639ed359a0517" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44449 } } peers { permanent_uuid: "0cbcf73bcb4741078c6b36d1c242fdcb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36639 } } peers { permanent_uuid: "770e74f0729f4f13b4346a856ef17522" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 40727 } }
I20260923 17:21:54.043432 1122 raft_consensus.cc:374] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:54.043582 1122 raft_consensus.cc:729] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770e74f0729f4f13b4346a856ef17522, State: Initialized, Role: FOLLOWER
I20260923 17:21:54.044392 1122 consensus_queue.cc:260] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8166903934c0892f639ed359a0517" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44449 } } peers { permanent_uuid: "0cbcf73bcb4741078c6b36d1c242fdcb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36639 } } peers { permanent_uuid: "770e74f0729f4f13b4346a856ef17522" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 40727 } }
I20260923 17:21:54.046274 1122 ts_tablet_manager.cc:1428] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260923 17:21:54.050410 1124 raft_consensus.cc:348] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8166903934c0892f639ed359a0517" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44449 } } peers { permanent_uuid: "0cbcf73bcb4741078c6b36d1c242fdcb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36639 } } peers { permanent_uuid: "770e74f0729f4f13b4346a856ef17522" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 40727 } }
I20260923 17:21:54.050921 1124 raft_consensus.cc:374] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:54.051018 1124 raft_consensus.cc:729] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cbcf73bcb4741078c6b36d1c242fdcb, State: Initialized, Role: FOLLOWER
I20260923 17:21:54.050861 1123 raft_consensus.cc:348] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8166903934c0892f639ed359a0517" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44449 } } peers { permanent_uuid: "0cbcf73bcb4741078c6b36d1c242fdcb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36639 } } peers { permanent_uuid: "770e74f0729f4f13b4346a856ef17522" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 40727 } }
I20260923 17:21:54.051383 1123 raft_consensus.cc:374] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:54.051496 1123 raft_consensus.cc:729] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef8166903934c0892f639ed359a0517, State: Initialized, Role: FOLLOWER
I20260923 17:21:54.051708 1124 consensus_queue.cc:260] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8166903934c0892f639ed359a0517" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44449 } } peers { permanent_uuid: "0cbcf73bcb4741078c6b36d1c242fdcb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36639 } } peers { permanent_uuid: "770e74f0729f4f13b4346a856ef17522" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 40727 } }
I20260923 17:21:54.052130 1123 consensus_queue.cc:260] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8166903934c0892f639ed359a0517" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44449 } } peers { permanent_uuid: "0cbcf73bcb4741078c6b36d1c242fdcb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36639 } } peers { permanent_uuid: "770e74f0729f4f13b4346a856ef17522" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 40727 } }
I20260923 17:21:54.052942 1103 heartbeater.cc:503] Master 127.28.72.254:42103 was elected leader, sending a full tablet report...
I20260923 17:21:54.053292 970 heartbeater.cc:503] Master 127.28.72.254:42103 was elected leader, sending a full tablet report...
I20260923 17:21:54.053599 1124 ts_tablet_manager.cc:1428] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260923 17:21:54.053798 1123 ts_tablet_manager.cc:1428] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260923 17:21:54.115851 838 tablet.cc:2369] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:21:54.139868 971 tablet.cc:2369] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:21:54.161880 1128 raft_consensus.cc:482] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:21:54.162214 1128 raft_consensus.cc:504] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8166903934c0892f639ed359a0517" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44449 } } peers { permanent_uuid: "0cbcf73bcb4741078c6b36d1c242fdcb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36639 } } peers { permanent_uuid: "770e74f0729f4f13b4346a856ef17522" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 40727 } }
I20260923 17:21:54.164094 1128 leader_election.cc:288] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ef8166903934c0892f639ed359a0517 (127.28.72.194:44449), 0cbcf73bcb4741078c6b36d1c242fdcb (127.28.72.195:36639)
W20260923 17:21:54.164276 1104 tablet.cc:2369] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:21:54.171834 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:54.165915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:50629 (local address 127.28.72.195:36639)
0923 17:21:54.166133 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:54.166140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:54.166581 (+ 441us) server_negotiation.cc:407] Connection header received
0923 17:21:54.166795 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:54.166803 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:54.166858 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:54.166953 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:54.167539 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.168215 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.169969 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.170823 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.170995 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.171074 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:54.171436 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 17:21:54.171567 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:21:54.171833 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:54.164609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:51299 (local address 127.28.72.194:44449)
0923 17:21:54.164822 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:54.164828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:54.165710 (+ 882us) server_negotiation.cc:407] Connection header received
0923 17:21:54.165946 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:54.165953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:54.166034 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:54.166150 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:54.166938 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.167896 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.169802 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.170684 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.170882 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.170957 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:54.171437 (+ 480us) server_negotiation.cc:299] Negotiation successful
0923 17:21:54.171559 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:21:54.172010 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:54.164852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:44449 (local address 127.28.72.193:51299)
0923 17:21:54.165535 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:54.165556 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:54.165710 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:54.166263 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:54.166269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:54.166326 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:54.166761 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:54.166771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.168029 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.168034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:54.169645 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:54.169657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.170893 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.170899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:54.171170 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.171207 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 17:21:54.171408 (+ 201us) client_negotiation.cc:240] Negotiation successful
0923 17:21:54.171551 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":65,"thread_start_us":366,"threads_started":1}
I20260923 17:21:54.172070 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:54.165763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36639 (local address 127.28.72.193:50629)
0923 17:21:54.166441 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:54.166460 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:54.166646 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:54.166988 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:54.166992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:54.167015 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:54.167391 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:54.167404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.168351 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.168356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:54.169821 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:54.169832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.171058 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.171065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:54.171275 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.171298 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:21:54.171406 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 17:21:54.171591 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":3,"thread_start_us":316,"threads_started":1}
I20260923 17:21:54.173183 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3a2e278002418a97498069bdc199c8" candidate_uuid: "770e74f0729f4f13b4346a856ef17522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cbcf73bcb4741078c6b36d1c242fdcb" is_pre_election: true
I20260923 17:21:54.173182 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3a2e278002418a97498069bdc199c8" candidate_uuid: "770e74f0729f4f13b4346a856ef17522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ef8166903934c0892f639ed359a0517" is_pre_election: true
I20260923 17:21:54.173777 1058 raft_consensus.cc:2454] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770e74f0729f4f13b4346a856ef17522 in term 0.
I20260923 17:21:54.173777 925 raft_consensus.cc:2454] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770e74f0729f4f13b4346a856ef17522 in term 0.
I20260923 17:21:54.174616 727 leader_election.cc:302] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ef8166903934c0892f639ed359a0517, 770e74f0729f4f13b4346a856ef17522; no voters:
I20260923 17:21:54.175005 1128 raft_consensus.cc:2781] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:21:54.175189 1128 raft_consensus.cc:482] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:21:54.175305 1128 raft_consensus.cc:3037] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:54.177484 1128 raft_consensus.cc:504] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8166903934c0892f639ed359a0517" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44449 } } peers { permanent_uuid: "0cbcf73bcb4741078c6b36d1c242fdcb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36639 } } peers { permanent_uuid: "770e74f0729f4f13b4346a856ef17522" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 40727 } }
I20260923 17:21:54.178077 1128 leader_election.cc:288] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [CANDIDATE]: Term 1 election: Requested vote from peers 1ef8166903934c0892f639ed359a0517 (127.28.72.194:44449), 0cbcf73bcb4741078c6b36d1c242fdcb (127.28.72.195:36639)
I20260923 17:21:54.178555 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3a2e278002418a97498069bdc199c8" candidate_uuid: "770e74f0729f4f13b4346a856ef17522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ef8166903934c0892f639ed359a0517"
I20260923 17:21:54.178597 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3a2e278002418a97498069bdc199c8" candidate_uuid: "770e74f0729f4f13b4346a856ef17522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cbcf73bcb4741078c6b36d1c242fdcb"
I20260923 17:21:54.178779 925 raft_consensus.cc:3037] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:54.178788 1058 raft_consensus.cc:3037] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:54.181141 1058 raft_consensus.cc:2454] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770e74f0729f4f13b4346a856ef17522 in term 1.
I20260923 17:21:54.181252 925 raft_consensus.cc:2454] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770e74f0729f4f13b4346a856ef17522 in term 1.
I20260923 17:21:54.181581 727 leader_election.cc:302] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cbcf73bcb4741078c6b36d1c242fdcb, 770e74f0729f4f13b4346a856ef17522; no voters:
I20260923 17:21:54.181963 1128 raft_consensus.cc:2781] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:54.182550 1128 raft_consensus.cc:686] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [term 1 LEADER]: Becoming Leader. State: Replica: 770e74f0729f4f13b4346a856ef17522, State: Running, Role: LEADER
I20260923 17:21:54.183179 1128 consensus_queue.cc:237] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8166903934c0892f639ed359a0517" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44449 } } peers { permanent_uuid: "0cbcf73bcb4741078c6b36d1c242fdcb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36639 } } peers { permanent_uuid: "770e74f0729f4f13b4346a856ef17522" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 40727 } }
I20260923 17:21:54.188237 649 catalog_manager.cc:5495] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 reported cstate change: term changed from 0 to 1, leader changed from <none> to 770e74f0729f4f13b4346a856ef17522 (127.28.72.193). New cstate: current_term: 1 leader_uuid: "770e74f0729f4f13b4346a856ef17522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8166903934c0892f639ed359a0517" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cbcf73bcb4741078c6b36d1c242fdcb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "770e74f0729f4f13b4346a856ef17522" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 40727 } health_report { overall_health: HEALTHY } } }
I20260923 17:21:54.244500 1121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:54.239159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58156 (local address 127.28.72.193:40727)
0923 17:21:54.239351 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:54.239357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:54.239458 (+ 101us) server_negotiation.cc:407] Connection header received
0923 17:21:54.239638 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:54.239643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:54.239752 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:54.239915 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:54.240279 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.240965 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.242495 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.242893 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.242986 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.243493 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:54.244048 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:54.244158 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:54.244235 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 17:21:54.244301 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:21:54.252754 1058 raft_consensus.cc:1261] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb [term 1 FOLLOWER]: Refusing update from remote peer 770e74f0729f4f13b4346a856ef17522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:54.252976 925 raft_consensus.cc:1261] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517 [term 1 FOLLOWER]: Refusing update from remote peer 770e74f0729f4f13b4346a856ef17522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:54.253777 1133 consensus_queue.cc:1035] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cbcf73bcb4741078c6b36d1c242fdcb" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:54.254158 1128 consensus_queue.cc:1035] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ef8166903934c0892f639ed359a0517" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:54.267326 1136 mvcc.cc:205] Tried to move back new op lower bound from 7332594131969523712 to 7332594131698647040. Current Snapshot: MvccSnapshot[applied={T|T < 7332594131969523712 or (T in {7332594131969523712})}]
I20260923 17:21:54.268726 1138 mvcc.cc:205] Tried to move back new op lower bound from 7332594131969523712 to 7332594131698647040. Current Snapshot: MvccSnapshot[applied={T|T < 7332594131969523712}]
I20260923 17:21:54.280083 649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:54.280313 649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:54.284315 649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 314c38c84c264892b7f414db30a9c9f4: Sending DeleteTablet for 3 replicas of tablet 0b3a2e278002418a97498069bdc199c8
I20260923 17:21:54.285694 905 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b3a2e278002418a97498069bdc199c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:54 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:45446
I20260923 17:21:54.285794 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b3a2e278002418a97498069bdc199c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:54 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:52942
I20260923 17:21:54.286043 772 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b3a2e278002418a97498069bdc199c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:54 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:58150
I20260923 17:21:54.289081 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 710
I20260923 17:21:54.298082 1147 tablet_replica.cc:330] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb: stopping tablet replica
I20260923 17:21:54.298629 1148 tablet_replica.cc:330] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522: stopping tablet replica
I20260923 17:21:54.298636 1147 raft_consensus.cc:2229] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:54.299043 1148 raft_consensus.cc:2229] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:21:54.299221 1147 raft_consensus.cc:2258] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:54.299816 1148 raft_consensus.cc:2258] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:54.302563 1148 ts_tablet_manager.cc:1905] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:54.304129 1146 tablet_replica.cc:330] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517: stopping tablet replica
I20260923 17:21:54.304597 1146 raft_consensus.cc:2229] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:54.305166 1146 raft_consensus.cc:2258] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:54.308485 1147 ts_tablet_manager.cc:1905] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:54.308820 1146 ts_tablet_manager.cc:1905] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:54.310949 1148 ts_tablet_manager.cc:1918] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:21:54.311162 1148 log.cc:1196] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/0b3a2e278002418a97498069bdc199c8
I20260923 17:21:54.311812 1148 ts_tablet_manager.cc:1939] T 0b3a2e278002418a97498069bdc199c8 P 770e74f0729f4f13b4346a856ef17522: Deleting consensus metadata
Sep 23 17:21:54 dist-test-slave-j0m4 krb5kdc[605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:21:54.313791 635 catalog_manager.cc:4841] TS 770e74f0729f4f13b4346a856ef17522 (127.28.72.193:40727): tablet 0b3a2e278002418a97498069bdc199c8 (table test-table [id=dbd2a9c6015a4e53866795fdbf582ca9]) successfully deleted
I20260923 17:21:54.318985 1146 ts_tablet_manager.cc:1918] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:21:54.319330 1146 log.cc:1196] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/0b3a2e278002418a97498069bdc199c8
I20260923 17:21:54.319953 1146 ts_tablet_manager.cc:1939] T 0b3a2e278002418a97498069bdc199c8 P 1ef8166903934c0892f639ed359a0517: Deleting consensus metadata
I20260923 17:21:54.320462 1147 ts_tablet_manager.cc:1918] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:21:54.320653 1147 log.cc:1196] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/0b3a2e278002418a97498069bdc199c8
I20260923 17:21:54.321201 1147 ts_tablet_manager.cc:1939] T 0b3a2e278002418a97498069bdc199c8 P 0cbcf73bcb4741078c6b36d1c242fdcb: Deleting consensus metadata
I20260923 17:21:54.322319 635 catalog_manager.cc:4841] TS 1ef8166903934c0892f639ed359a0517 (127.28.72.194:44449): tablet 0b3a2e278002418a97498069bdc199c8 (table test-table [id=dbd2a9c6015a4e53866795fdbf582ca9]) successfully deleted
I20260923 17:21:54.323101 635 catalog_manager.cc:4841] TS 0cbcf73bcb4741078c6b36d1c242fdcb (127.28.72.195:36639): tablet 0b3a2e278002418a97498069bdc199c8 (table test-table [id=dbd2a9c6015a4e53866795fdbf582ca9]) successfully deleted
I20260923 17:21:54.324074 1121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:54.289914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.28.72.193:40727)
0923 17:21:54.290141 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:54.290149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:54.303324 (+ 13175us) server_negotiation.cc:407] Connection header received
0923 17:21:54.303550 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:54.303555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:54.303650 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:54.303761 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:54.305709 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.306861 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.308787 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.309549 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.309648 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.315102 (+ 5454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:54.315145 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:54.315166 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:54.315224 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:54.319437 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:54.320576 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:54.320593 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:54.320596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:54.320806 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:54.321301 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:54.321309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:54.321312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:54.321822 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:54.322147 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:54.323411 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0923 17:21:54.323699 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:21:54.324882 832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:54.424386 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 710
I20260923 17:21:54.439308 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 843
Sep 23 17:21:54 dist-test-slave-j0m4 krb5kdc[605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:21:54.455680 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:54.440101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.28.72.194:44449)
0923 17:21:54.440309 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:54.440314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:54.440335 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:21:54.440460 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:54.440464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:54.440532 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:54.440628 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:54.442028 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.443083 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.444621 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.445051 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.445165 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.449350 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:54.449437 (+ 87us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:54.449454 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:54.449502 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:54.452614 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:54.453447 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:54.453455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:54.453457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:54.453622 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:54.454004 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:54.454008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:54.454011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:54.454523 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:54.454740 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:54.455074 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 17:21:54.455382 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 17:21:54.456410 965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:54.549171 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 843
I20260923 17:21:54.566951 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 976
Sep 23 17:21:54 dist-test-slave-j0m4 krb5kdc[605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:21:54.583938 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:54.567830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.28.72.195:36639)
0923 17:21:54.568067 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:54.568072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:54.568093 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:21:54.568172 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:54.568176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:54.568248 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:54.568331 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:54.569645 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.570492 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.572358 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.572793 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.572909 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.577659 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:54.577677 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:54.577693 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:54.577736 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:54.580879 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:54.581776 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:54.581784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:54.581787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:54.581943 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:54.582445 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:54.582449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:54.582451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:54.582827 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:54.583051 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:54.583495 (+ 444us) server_negotiation.cc:299] Negotiation successful
0923 17:21:54.583675 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260923 17:21:54.584573 1098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:54.675050 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 976
I20260923 17:21:54.691746 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 618
I20260923 17:21:54.692978 679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:54.801617 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 618
2026-09-23T17:21:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4353 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T17:21:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:54Z Disabled control of system clock
I20260923 17:21:54.873737 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:37169
--webserver_interface=127.28.72.254
--webserver_port=0
--builtin_ntp_servers=127.28.72.212:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:37169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 17:21:55.207429 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:55.207751 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:55.207913 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:55.215497 1157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:55.215623 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:55.215696 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:55.215824 1157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:55.215893 1157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:21:55.225317 1157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:37169
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:37169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:55.226744 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:55.228685 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:55.241129 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:55.241202 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:55.241168 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:55.242357 1157 server_base.cc:1034] running on GCE node
I20260923 17:21:55.243271 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:55.244747 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:55.245973 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790184115245904 us; error 60 us; skew 500 ppm
I20260923 17:21:55.246338 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:21:55.248678 1157 webserver.cc:460] Webserver started at http://127.28.72.254:33853/ using document root <none> and password file <none>
I20260923 17:21:55.249402 1157 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:55.249518 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:55.249863 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:55.252889 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "38cbb397218c4e6893eb57c99daff214"
format_stamp: "Formatted at 2026-09-23 17:21:55 on dist-test-slave-j0m4"
server_key: "d12ad2bef0be024c28b6c702b5bcd8a7"
server_key_iv: "200a505cb1846b51e0599012c1814015"
server_key_version: "encryptionkey@0"
I20260923 17:21:55.253715 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "38cbb397218c4e6893eb57c99daff214"
format_stamp: "Formatted at 2026-09-23 17:21:55 on dist-test-slave-j0m4"
server_key: "d12ad2bef0be024c28b6c702b5bcd8a7"
server_key_iv: "200a505cb1846b51e0599012c1814015"
server_key_version: "encryptionkey@0"
I20260923 17:21:55.259191 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260923 17:21:55.262732 1171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:55.264581 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 17:21:55.264786 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "38cbb397218c4e6893eb57c99daff214"
format_stamp: "Formatted at 2026-09-23 17:21:55 on dist-test-slave-j0m4"
server_key: "d12ad2bef0be024c28b6c702b5bcd8a7"
server_key_iv: "200a505cb1846b51e0599012c1814015"
server_key_version: "encryptionkey@0"
I20260923 17:21:55.265041 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:55.272027 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:55.273051 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:55.273439 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:55.294443 1157 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:37169
I20260923 17:21:55.294492 1222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:37169 every 8 connection(s)
I20260923 17:21:55.295990 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:55.299906 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:55.304104 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 1157
I20260923 17:21:55.304342 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:21:55.305111 28963 external_mini_cluster.cc:1428] Setting key fb00f894da942866029ced289f96f28d
I20260923 17:21:55.311812 1223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214: Bootstrap starting.
I20260923 17:21:55.313773 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:55.307358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43758 (local address 127.28.72.254:37169)
0923 17:21:55.308390 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:55.308401 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:55.308457 (+ 56us) server_negotiation.cc:407] Connection header received
0923 17:21:55.310179 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:55.310204 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:55.310645 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:55.310987 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:55.312142 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:55.312148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:55.312161 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:55.312183 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:55.312242 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:55.312258 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:55.312441 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:55.312542 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 17:21:55.312812 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":442,"threads_started":1}
I20260923 17:21:55.315397 1223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:55.316471 1223 log.cc:824] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:55.319476 1223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214: No bootstrap required, opened a new log
I20260923 17:21:55.328207 1223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38cbb397218c4e6893eb57c99daff214" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37169 } }
I20260923 17:21:55.328661 1223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:55.328783 1223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38cbb397218c4e6893eb57c99daff214, State: Initialized, Role: FOLLOWER
I20260923 17:21:55.329399 1223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38cbb397218c4e6893eb57c99daff214" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37169 } }
I20260923 17:21:55.329662 1223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:55.329836 1223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:55.330035 1223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:55.332552 1223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38cbb397218c4e6893eb57c99daff214" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37169 } }
I20260923 17:21:55.333164 1223 leader_election.cc:302] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38cbb397218c4e6893eb57c99daff214; no voters:
I20260923 17:21:55.333899 1223 leader_election.cc:288] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:55.334053 1228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:55.334842 1228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [term 1 LEADER]: Becoming Leader. State: Replica: 38cbb397218c4e6893eb57c99daff214, State: Running, Role: LEADER
I20260923 17:21:55.335561 1228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38cbb397218c4e6893eb57c99daff214" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37169 } }
I20260923 17:21:55.336362 1223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:55.338450 1230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38cbb397218c4e6893eb57c99daff214. Latest consensus state: current_term: 1 leader_uuid: "38cbb397218c4e6893eb57c99daff214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38cbb397218c4e6893eb57c99daff214" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37169 } } }
I20260923 17:21:55.338579 1229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38cbb397218c4e6893eb57c99daff214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38cbb397218c4e6893eb57c99daff214" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37169 } } }
I20260923 17:21:55.338893 1230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:55.338897 1229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214 [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:55.341398 1235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:55.346958 1235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:55.357678 1235 catalog_manager.cc:1347] Generated new cluster ID: 79f55153cb3641f0954396db985b7fe6
I20260923 17:21:55.357811 1235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:55.370633 1235 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:55.371174 1235 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:55.378979 1235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214: Generated new TSK 0
I20260923 17:21:55.380191 1235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:21:55.386379 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37169
--builtin_ntp_servers=127.28.72.212:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 17:21:55.730465 1247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:55.730767 1247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:55.730858 1247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:55.738462 1247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:55.738613 1247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:55.738732 1247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:21:55.748672 1247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:55.749959 1247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:55.751754 1247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:55.765671 1252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:55.765821 1253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:55.766089 1255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:55.766947 1247 server_base.cc:1034] running on GCE node
I20260923 17:21:55.767673 1247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:55.768690 1247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:55.769985 1247 hybrid_clock.cc:648] HybridClock initialized: now 1790184115769926 us; error 48 us; skew 500 ppm
I20260923 17:21:55.770349 1247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:21:55.772939 1247 webserver.cc:460] Webserver started at http://127.28.72.193:34595/ using document root <none> and password file <none>
I20260923 17:21:55.773691 1247 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:55.773835 1247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:55.774183 1247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:55.777245 1247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "b735604244554b9991b4925782b8e0f0"
format_stamp: "Formatted at 2026-09-23 17:21:55 on dist-test-slave-j0m4"
server_key: "ece4957d170119391cefdd2969fb2a23"
server_key_iv: "d8a60da0a5b2958082c7cb2180281fa7"
server_key_version: "encryptionkey@0"
I20260923 17:21:55.778086 1247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "b735604244554b9991b4925782b8e0f0"
format_stamp: "Formatted at 2026-09-23 17:21:55 on dist-test-slave-j0m4"
server_key: "ece4957d170119391cefdd2969fb2a23"
server_key_iv: "d8a60da0a5b2958082c7cb2180281fa7"
server_key_version: "encryptionkey@0"
I20260923 17:21:55.783499 1247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:21:55.787317 1261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:55.788888 1247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:21:55.789108 1247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "b735604244554b9991b4925782b8e0f0"
format_stamp: "Formatted at 2026-09-23 17:21:55 on dist-test-slave-j0m4"
server_key: "ece4957d170119391cefdd2969fb2a23"
server_key_iv: "d8a60da0a5b2958082c7cb2180281fa7"
server_key_version: "encryptionkey@0"
I20260923 17:21:55.789352 1247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:55.795840 1247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:55.796757 1247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:55.797063 1247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:55.798110 1247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:55.799779 1247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:55.799906 1247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:55.800052 1247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:55.800140 1247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:55.835727 1247 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:43529
I20260923 17:21:55.835790 1373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:43529 every 8 connection(s)
I20260923 17:21:55.837199 1247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:21:55.846354 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:55.840078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:39781 (local address 127.28.72.254:37169)
0923 17:21:55.840713 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:55.840717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:55.841971 (+ 1254us) server_negotiation.cc:407] Connection header received
0923 17:21:55.843859 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:55.843864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:55.843954 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:55.844082 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:55.845536 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:55.845542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:55.845547 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:55.845570 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:55.845616 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:55.845628 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:55.845757 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:55.845982 (+ 225us) server_negotiation.cc:299] Negotiation successful
0923 17:21:55.846055 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":279,"threads_started":1}
I20260923 17:21:55.846949 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 1247
I20260923 17:21:55.847229 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:21:55.847491 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:55.840791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37169 (local address 127.28.72.193:39781)
0923 17:21:55.841797 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:55.841839 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:55.843564 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:55.844352 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:55.844361 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:55.844485 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:55.845220 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:21:55.845222 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:55.845262 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:21:55.845355 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:55.845726 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:55.845745 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:55.845752 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:21:55.846069 (+ 317us) client_negotiation.cc:240] Negotiation successful
0923 17:21:55.846355 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":348,"threads_started":1}
I20260923 17:21:55.848064 28963 external_mini_cluster.cc:1428] Setting key c6cebf573d2b331336c5f70343d10009
I20260923 17:21:55.850076 1374 heartbeater.cc:347] Connected to a master server at 127.28.72.254:37169
I20260923 17:21:55.850626 1374 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:55.851451 1374 heartbeater.cc:511] Master 127.28.72.254:37169 requested a full tablet report, sending...
I20260923 17:21:55.851871 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37169
--builtin_ntp_servers=127.28.72.212:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 17:21:55.854105 1188 ts_manager.cc:194] Registered new tserver with Master: b735604244554b9991b4925782b8e0f0 (127.28.72.193:43529)
W20260923 17:21:56.212057 1378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:56.212369 1378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:56.212514 1378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:56.220126 1378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:56.220249 1378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:56.220428 1378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:21:56.230461 1378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:56.231858 1378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:56.233700 1378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:56.248253 1383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:56.248374 1384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:56.248509 1386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:56.249130 1378 server_base.cc:1034] running on GCE node
I20260923 17:21:56.249788 1378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:56.250741 1378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:56.251959 1378 hybrid_clock.cc:648] HybridClock initialized: now 1790184116251937 us; error 53 us; skew 500 ppm
I20260923 17:21:56.252240 1378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:21:56.254675 1378 webserver.cc:460] Webserver started at http://127.28.72.194:35021/ using document root <none> and password file <none>
I20260923 17:21:56.255512 1378 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:56.255607 1378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:56.255899 1378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:56.258908 1378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "bb91e016fd844be18687062ddf7ff629"
format_stamp: "Formatted at 2026-09-23 17:21:56 on dist-test-slave-j0m4"
server_key: "f4b3fd48380c8c7f24723b396e7553dc"
server_key_iv: "b9bd315aba48598f7d9da64f6cee0ab5"
server_key_version: "encryptionkey@0"
I20260923 17:21:56.259774 1378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "bb91e016fd844be18687062ddf7ff629"
format_stamp: "Formatted at 2026-09-23 17:21:56 on dist-test-slave-j0m4"
server_key: "f4b3fd48380c8c7f24723b396e7553dc"
server_key_iv: "b9bd315aba48598f7d9da64f6cee0ab5"
server_key_version: "encryptionkey@0"
I20260923 17:21:56.265390 1378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:21:56.269512 1392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:56.271593 1378 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:21:56.271766 1378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "bb91e016fd844be18687062ddf7ff629"
format_stamp: "Formatted at 2026-09-23 17:21:56 on dist-test-slave-j0m4"
server_key: "f4b3fd48380c8c7f24723b396e7553dc"
server_key_iv: "b9bd315aba48598f7d9da64f6cee0ab5"
server_key_version: "encryptionkey@0"
I20260923 17:21:56.271991 1378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:56.278535 1378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:56.279434 1378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:56.279685 1378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:56.280681 1378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:56.282191 1378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:56.282282 1378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:56.282361 1378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:56.282418 1378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:56.318017 1378 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:34109
I20260923 17:21:56.318074 1504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:34109 every 8 connection(s)
I20260923 17:21:56.319466 1378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:21:56.327890 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 1378
I20260923 17:21:56.328136 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:21:56.328687 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.322977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:52881 (local address 127.28.72.254:37169)
0923 17:21:56.323226 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:56.323230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:56.324472 (+ 1242us) server_negotiation.cc:407] Connection header received
0923 17:21:56.326210 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:56.326216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:56.326296 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:56.326409 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:56.327914 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:56.327928 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:56.327933 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:56.327959 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:56.327990 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:56.328001 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:56.328137 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:56.328409 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 17:21:56.328462 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:21:56.328711 28963 external_mini_cluster.cc:1428] Setting key de99d7621226a6550e581113445f79f6
I20260923 17:21:56.330083 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.323345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37169 (local address 127.28.72.194:52881)
0923 17:21:56.324305 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:56.324341 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:56.325918 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:56.326716 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:56.326725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:56.326822 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:56.327606 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:21:56.327608 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:56.327649 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:21:56.327736 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:56.328151 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:56.328166 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:56.328177 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 17:21:56.328512 (+ 335us) client_negotiation.cc:240] Negotiation successful
0923 17:21:56.328796 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":279,"threads_started":1}
I20260923 17:21:56.331497 1505 heartbeater.cc:347] Connected to a master server at 127.28.72.254:37169
I20260923 17:21:56.331818 1505 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:56.332072 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37169
--builtin_ntp_servers=127.28.72.212:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 17:21:56.332494 1505 heartbeater.cc:511] Master 127.28.72.254:37169 requested a full tablet report, sending...
I20260923 17:21:56.333928 1188 ts_manager.cc:194] Registered new tserver with Master: bb91e016fd844be18687062ddf7ff629 (127.28.72.194:34109)
W20260923 17:21:56.672729 1508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:56.673054 1508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:56.673199 1508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:56.681490 1508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:56.681624 1508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:56.681805 1508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:21:56.691774 1508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:56.693249 1508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:56.695235 1508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:56.709555 1514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:56.709584 1513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:56.710263 1516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:56.711225 1508 server_base.cc:1034] running on GCE node
I20260923 17:21:56.711918 1508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:56.713017 1508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:56.714299 1508 hybrid_clock.cc:648] HybridClock initialized: now 1790184116714215 us; error 65 us; skew 500 ppm
I20260923 17:21:56.714661 1508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:21:56.717185 1508 webserver.cc:460] Webserver started at http://127.28.72.195:38769/ using document root <none> and password file <none>
I20260923 17:21:56.717935 1508 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:56.718067 1508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:56.718420 1508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:56.721565 1508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "02f60545f5654e68937796bb3ba7e238"
format_stamp: "Formatted at 2026-09-23 17:21:56 on dist-test-slave-j0m4"
server_key: "d47bfa8f45127b1d85ee8f276d7fbf74"
server_key_iv: "d5e51f1728d7e3a9e8510ef377fbb9aa"
server_key_version: "encryptionkey@0"
I20260923 17:21:56.722463 1508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "02f60545f5654e68937796bb3ba7e238"
format_stamp: "Formatted at 2026-09-23 17:21:56 on dist-test-slave-j0m4"
server_key: "d47bfa8f45127b1d85ee8f276d7fbf74"
server_key_iv: "d5e51f1728d7e3a9e8510ef377fbb9aa"
server_key_version: "encryptionkey@0"
I20260923 17:21:56.728358 1508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:21:56.732502 1522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:56.734462 1508 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:21:56.734702 1508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "02f60545f5654e68937796bb3ba7e238"
format_stamp: "Formatted at 2026-09-23 17:21:56 on dist-test-slave-j0m4"
server_key: "d47bfa8f45127b1d85ee8f276d7fbf74"
server_key_iv: "d5e51f1728d7e3a9e8510ef377fbb9aa"
server_key_version: "encryptionkey@0"
I20260923 17:21:56.734953 1508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:56.741238 1508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:56.742422 1508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:56.742741 1508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:56.743870 1508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:56.745472 1508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:56.745603 1508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:56.745738 1508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:56.745822 1508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:56.781759 1508 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:33629
I20260923 17:21:56.781821 1634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:33629 every 8 connection(s)
I20260923 17:21:56.783257 1508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:21:56.783872 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 1508
I20260923 17:21:56.784094 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:21:56.784749 28963 external_mini_cluster.cc:1428] Setting key fe51d0a56f385137afc4a50d4755955e
I20260923 17:21:56.792100 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.786155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:49305 (local address 127.28.72.254:37169)
0923 17:21:56.786340 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:56.786349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:56.788065 (+ 1716us) server_negotiation.cc:407] Connection header received
0923 17:21:56.789719 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:56.789723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:56.789775 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:56.789870 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:56.791393 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:56.791398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:56.791401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:56.791420 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:56.791452 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:56.791464 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:56.791591 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:56.791802 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 17:21:56.791858 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:21:56.793231 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.786761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37169 (local address 127.28.72.195:49305)
0923 17:21:56.787914 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:56.787960 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:56.789456 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:56.790196 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:56.790205 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:56.790310 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:56.791014 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:21:56.791017 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:56.791059 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:21:56.791216 (+ 157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:56.791573 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:56.791581 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:56.791587 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:21:56.791826 (+ 239us) client_negotiation.cc:240] Negotiation successful
0923 17:21:56.792083 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":376,"threads_started":1}
I20260923 17:21:56.794823 1635 heartbeater.cc:347] Connected to a master server at 127.28.72.254:37169
I20260923 17:21:56.795215 1635 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:56.795855 1635 heartbeater.cc:511] Master 127.28.72.254:37169 requested a full tablet report, sending...
I20260923 17:21:56.797389 1188 ts_manager.cc:194] Registered new tserver with Master: 02f60545f5654e68937796bb3ba7e238 (127.28.72.195:33629)
I20260923 17:21:56.800576 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:21:56.808007 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.805350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.28.72.254:37169)
0923 17:21:56.805598 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:56.805606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:56.805903 (+ 297us) server_negotiation.cc:407] Connection header received
0923 17:21:56.806072 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:56.806076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:56.806130 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:56.806206 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:56.807218 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:56.807223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:56.807227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:56.807244 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:56.807279 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:56.807295 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:56.807420 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:56.807606 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 17:21:56.807704 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 17:21:56.812541 1188 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:21:56.819366 1188 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:21:56.852684 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.832555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.28.72.194:34109)
0923 17:21:56.848412 (+ 15857us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:56.848426 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:56.848474 (+ 48us) server_negotiation.cc:407] Connection header received
0923 17:21:56.848574 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:56.848587 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:56.848908 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:56.849165 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:56.851354 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:56.851367 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:56.851386 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:56.851422 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:56.851486 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:56.851511 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:56.851633 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:56.852059 (+ 426us) server_negotiation.cc:299] Negotiation successful
0923 17:21:56.852202 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15594,"thread_start_us":17441,"threads_started":1}
I20260923 17:21:56.853900 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.832288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:34109 (local address 127.0.0.1:58238)
0923 17:21:56.845771 (+ 13483us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:56.845852 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:56.846085 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:56.849226 (+ 3141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:56.849239 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:56.849400 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:56.850953 (+ 1553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:21:56.850956 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:56.850986 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:21:56.851100 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:56.851745 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:56.851762 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:56.851774 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 17:21:56.853155 (+ 1381us) client_negotiation.cc:240] Negotiation successful
0923 17:21:56.853287 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13093,"thread_start_us":18336,"threads_started":1}
I20260923 17:21:56.856078 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.832418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:43529 (local address 127.0.0.1:43304)
0923 17:21:56.851032 (+ 18614us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:56.851052 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:56.851205 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:56.853176 (+ 1971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:56.853185 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:56.853223 (+ 38us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:56.854247 (+ 1024us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:21:56.854254 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:56.854292 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:21:56.854357 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:56.855361 (+ 1004us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:56.855366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:56.855372 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:21:56.855551 (+ 179us) client_negotiation.cc:240] Negotiation successful
0923 17:21:56.855604 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18433,"thread_start_us":13856,"threads_started":1}
I20260923 17:21:56.857234 1650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.832693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.28.72.193:43529)
0923 17:21:56.850711 (+ 18018us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:56.850719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:56.851354 (+ 635us) server_negotiation.cc:407] Connection header received
0923 17:21:56.851465 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:56.851479 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:56.851805 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:56.851993 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:56.854909 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:56.854921 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:56.854941 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:56.854966 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:56.855038 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:56.855064 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:56.855319 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:56.856777 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0923 17:21:56.856890 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17861,"thread_start_us":17674,"threads_started":1}
I20260923 17:21:56.857487 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.851158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:33629 (local address 127.0.0.1:53724)
0923 17:21:56.854339 (+ 3181us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:56.854365 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:56.854470 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:56.855758 (+ 1288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:56.855763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:56.855808 (+ 45us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:56.856470 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:21:56.856475 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:56.856495 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:21:56.856540 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:56.857078 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:56.857083 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:56.857088 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:21:56.857180 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 17:21:56.857228 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3011,"thread_start_us":4022,"threads_started":1}
I20260923 17:21:56.857659 1652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.851522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.28.72.195:33629)
0923 17:21:56.855162 (+ 3640us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:56.855171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:56.855213 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:21:56.855291 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:56.855298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:56.855572 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:56.855777 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:56.856888 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:56.856902 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:56.856912 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:56.856925 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:56.856961 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:56.856973 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:56.857159 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:56.857221 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 17:21:56.857309 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3414,"thread_start_us":3243,"threads_started":1}
I20260923 17:21:56.858198 1374 heartbeater.cc:503] Master 127.28.72.254:37169 was elected leader, sending a full tablet report...
I20260923 17:21:56.860702 1440 tablet_service.cc:1467] Processing CreateTablet for tablet b9034f12ce564113bb6b14955d996d53 (DEFAULT_TABLE table=test-table [id=7c6ab53e39e54ceb9b83323fe25cd1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:56.860801 1309 tablet_service.cc:1467] Processing CreateTablet for tablet b9034f12ce564113bb6b14955d996d53 (DEFAULT_TABLE table=test-table [id=7c6ab53e39e54ceb9b83323fe25cd1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:56.862274 1440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9034f12ce564113bb6b14955d996d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:56.862267 1309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9034f12ce564113bb6b14955d996d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:56.862944 1570 tablet_service.cc:1467] Processing CreateTablet for tablet b9034f12ce564113bb6b14955d996d53 (DEFAULT_TABLE table=test-table [id=7c6ab53e39e54ceb9b83323fe25cd1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:56.864143 1570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9034f12ce564113bb6b14955d996d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:56.877815 1653 tablet_bootstrap.cc:492] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238: Bootstrap starting.
I20260923 17:21:56.882953 1653 tablet_bootstrap.cc:654] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:56.884600 1653 log.cc:824] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:56.886497 1655 tablet_bootstrap.cc:492] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0: Bootstrap starting.
I20260923 17:21:56.889213 1653 tablet_bootstrap.cc:492] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238: No bootstrap required, opened a new log
I20260923 17:21:56.889200 1654 tablet_bootstrap.cc:492] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629: Bootstrap starting.
I20260923 17:21:56.889611 1653 ts_tablet_manager.cc:1397] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260923 17:21:56.891397 1655 tablet_bootstrap.cc:654] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:56.892863 1655 log.cc:824] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:56.894203 1654 tablet_bootstrap.cc:654] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:56.895803 1654 log.cc:824] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:56.897049 1655 tablet_bootstrap.cc:492] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0: No bootstrap required, opened a new log
I20260923 17:21:56.897404 1655 ts_tablet_manager.cc:1397] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260923 17:21:56.899415 1654 tablet_bootstrap.cc:492] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629: No bootstrap required, opened a new log
I20260923 17:21:56.899823 1654 ts_tablet_manager.cc:1397] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 17:21:56.901681 1653 raft_consensus.cc:348] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb91e016fd844be18687062ddf7ff629" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34109 } } peers { permanent_uuid: "b735604244554b9991b4925782b8e0f0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43529 } } peers { permanent_uuid: "02f60545f5654e68937796bb3ba7e238" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 33629 } }
I20260923 17:21:56.902238 1653 raft_consensus.cc:374] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:56.902390 1653 raft_consensus.cc:729] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f60545f5654e68937796bb3ba7e238, State: Initialized, Role: FOLLOWER
I20260923 17:21:56.904168 1653 consensus_queue.cc:260] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb91e016fd844be18687062ddf7ff629" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34109 } } peers { permanent_uuid: "b735604244554b9991b4925782b8e0f0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43529 } } peers { permanent_uuid: "02f60545f5654e68937796bb3ba7e238" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 33629 } }
I20260923 17:21:56.905454 1635 heartbeater.cc:503] Master 127.28.72.254:37169 was elected leader, sending a full tablet report...
I20260923 17:21:56.906198 1653 ts_tablet_manager.cc:1428] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260923 17:21:56.908950 1655 raft_consensus.cc:348] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb91e016fd844be18687062ddf7ff629" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34109 } } peers { permanent_uuid: "b735604244554b9991b4925782b8e0f0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43529 } } peers { permanent_uuid: "02f60545f5654e68937796bb3ba7e238" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 33629 } }
I20260923 17:21:56.909461 1655 raft_consensus.cc:374] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:56.909569 1655 raft_consensus.cc:729] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b735604244554b9991b4925782b8e0f0, State: Initialized, Role: FOLLOWER
I20260923 17:21:56.910312 1655 consensus_queue.cc:260] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb91e016fd844be18687062ddf7ff629" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34109 } } peers { permanent_uuid: "b735604244554b9991b4925782b8e0f0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43529 } } peers { permanent_uuid: "02f60545f5654e68937796bb3ba7e238" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 33629 } }
I20260923 17:21:56.912209 1655 ts_tablet_manager.cc:1428] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260923 17:21:56.911936 1654 raft_consensus.cc:348] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb91e016fd844be18687062ddf7ff629" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34109 } } peers { permanent_uuid: "b735604244554b9991b4925782b8e0f0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43529 } } peers { permanent_uuid: "02f60545f5654e68937796bb3ba7e238" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 33629 } }
I20260923 17:21:56.912506 1654 raft_consensus.cc:374] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:56.912645 1654 raft_consensus.cc:729] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb91e016fd844be18687062ddf7ff629, State: Initialized, Role: FOLLOWER
I20260923 17:21:56.913287 1654 consensus_queue.cc:260] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb91e016fd844be18687062ddf7ff629" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34109 } } peers { permanent_uuid: "b735604244554b9991b4925782b8e0f0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43529 } } peers { permanent_uuid: "02f60545f5654e68937796bb3ba7e238" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 33629 } }
I20260923 17:21:56.914444 1505 heartbeater.cc:503] Master 127.28.72.254:37169 was elected leader, sending a full tablet report...
I20260923 17:21:56.915180 1654 ts_tablet_manager.cc:1428] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629: Time spent starting tablet: real 0.015s user 0.011s sys 0.000s
I20260923 17:21:57.035169 1661 raft_consensus.cc:482] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260923 17:21:57.035454 1636 tablet.cc:2369] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:21:57.035557 1661 raft_consensus.cc:504] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb91e016fd844be18687062ddf7ff629" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34109 } } peers { permanent_uuid: "b735604244554b9991b4925782b8e0f0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43529 } } peers { permanent_uuid: "02f60545f5654e68937796bb3ba7e238" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 33629 } }
I20260923 17:21:57.037500 1661 leader_election.cc:288] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b735604244554b9991b4925782b8e0f0 (127.28.72.193:43529), 02f60545f5654e68937796bb3ba7e238 (127.28.72.195:33629)
I20260923 17:21:57.041291 1650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.038042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:46497 (local address 127.28.72.193:43529)
0923 17:21:57.038442 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:57.038449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:57.039000 (+ 551us) server_negotiation.cc:407] Connection header received
0923 17:21:57.039098 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:57.039103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:57.039189 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:57.039300 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:57.040633 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:57.040641 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:57.040647 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:57.040670 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:57.040711 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:57.040727 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:57.040871 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:57.040970 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 17:21:57.041033 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260923 17:21:57.041404 1652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.038045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:38117 (local address 127.28.72.195:33629)
0923 17:21:57.038277 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:57.038285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:57.038798 (+ 513us) server_negotiation.cc:407] Connection header received
0923 17:21:57.039101 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:57.039106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:57.039189 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:57.039299 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:57.040737 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:57.040742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:57.040747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:57.040768 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:57.040806 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:57.040821 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:57.040936 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:57.041041 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 17:21:57.041092 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 17:21:57.041694 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.037913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:33629 (local address 127.28.72.194:38117)
0923 17:21:57.038665 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.038691 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:57.038912 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:57.039385 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:57.039393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:57.039421 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:57.040539 (+ 1118us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:21:57.040546 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:57.040563 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:21:57.040607 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:57.040915 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:57.040922 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:57.040930 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:21:57.041076 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 17:21:57.041111 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":345,"threads_started":1}
I20260923 17:21:57.041716 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.038057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:43529 (local address 127.28.72.194:46497)
0923 17:21:57.038889 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.038905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:57.038999 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:57.039416 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:57.039421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:57.039438 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:57.040266 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:21:57.040272 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:57.040293 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:21:57.040445 (+ 152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:57.040835 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:57.040840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:57.040845 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:21:57.041029 (+ 184us) client_negotiation.cc:240] Negotiation successful
0923 17:21:57.041097 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":553,"threads_started":1}
I20260923 17:21:57.042773 1590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9034f12ce564113bb6b14955d996d53" candidate_uuid: "bb91e016fd844be18687062ddf7ff629" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f60545f5654e68937796bb3ba7e238" is_pre_election: true
I20260923 17:21:57.042779 1329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9034f12ce564113bb6b14955d996d53" candidate_uuid: "bb91e016fd844be18687062ddf7ff629" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b735604244554b9991b4925782b8e0f0" is_pre_election: true
I20260923 17:21:57.043344 1590 raft_consensus.cc:2454] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb91e016fd844be18687062ddf7ff629 in term 0.
I20260923 17:21:57.043375 1329 raft_consensus.cc:2454] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb91e016fd844be18687062ddf7ff629 in term 0.
I20260923 17:21:57.044116 1393 leader_election.cc:302] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f60545f5654e68937796bb3ba7e238, bb91e016fd844be18687062ddf7ff629; no voters:
I20260923 17:21:57.044615 1661 raft_consensus.cc:2781] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:21:57.044759 1661 raft_consensus.cc:482] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:21:57.044870 1661 raft_consensus.cc:3037] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:57.047384 1661 raft_consensus.cc:504] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb91e016fd844be18687062ddf7ff629" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34109 } } peers { permanent_uuid: "b735604244554b9991b4925782b8e0f0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43529 } } peers { permanent_uuid: "02f60545f5654e68937796bb3ba7e238" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 33629 } }
I20260923 17:21:57.048134 1661 leader_election.cc:288] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [CANDIDATE]: Term 1 election: Requested vote from peers b735604244554b9991b4925782b8e0f0 (127.28.72.193:43529), 02f60545f5654e68937796bb3ba7e238 (127.28.72.195:33629)
I20260923 17:21:57.048691 1329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9034f12ce564113bb6b14955d996d53" candidate_uuid: "bb91e016fd844be18687062ddf7ff629" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b735604244554b9991b4925782b8e0f0"
I20260923 17:21:57.048811 1590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9034f12ce564113bb6b14955d996d53" candidate_uuid: "bb91e016fd844be18687062ddf7ff629" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f60545f5654e68937796bb3ba7e238"
I20260923 17:21:57.048945 1329 raft_consensus.cc:3037] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:57.049018 1590 raft_consensus.cc:3037] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:57.052459 1590 raft_consensus.cc:2454] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb91e016fd844be18687062ddf7ff629 in term 1.
I20260923 17:21:57.052459 1329 raft_consensus.cc:2454] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb91e016fd844be18687062ddf7ff629 in term 1.
I20260923 17:21:57.053030 1396 leader_election.cc:302] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02f60545f5654e68937796bb3ba7e238, b735604244554b9991b4925782b8e0f0, bb91e016fd844be18687062ddf7ff629; no voters:
I20260923 17:21:57.053521 1661 raft_consensus.cc:2781] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:57.054622 1661 raft_consensus.cc:686] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [term 1 LEADER]: Becoming Leader. State: Replica: bb91e016fd844be18687062ddf7ff629, State: Running, Role: LEADER
I20260923 17:21:57.055228 1661 consensus_queue.cc:237] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb91e016fd844be18687062ddf7ff629" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34109 } } peers { permanent_uuid: "b735604244554b9991b4925782b8e0f0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43529 } } peers { permanent_uuid: "02f60545f5654e68937796bb3ba7e238" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 33629 } }
I20260923 17:21:57.060710 1188 catalog_manager.cc:5495] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb91e016fd844be18687062ddf7ff629 (127.28.72.194). New cstate: current_term: 1 leader_uuid: "bb91e016fd844be18687062ddf7ff629" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb91e016fd844be18687062ddf7ff629" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 34109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b735604244554b9991b4925782b8e0f0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02f60545f5654e68937796bb3ba7e238" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 33629 } health_report { overall_health: UNKNOWN } } }
W20260923 17:21:57.071635 1506 tablet.cc:2369] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:21:57.090003 1375 tablet.cc:2369] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:21:57.110975 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.108617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.28.72.194:34109)
0923 17:21:57.108776 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:57.108781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:57.108843 (+ 62us) server_negotiation.cc:407] Connection header received
0923 17:21:57.109075 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:57.109080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:57.109165 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:57.109260 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:57.110232 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:57.110237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:57.110240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:57.110257 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:57.110288 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:57.110300 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:57.110481 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:57.110654 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 17:21:57.110708 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:21:57.120275 1590 raft_consensus.cc:1261] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238 [term 1 FOLLOWER]: Refusing update from remote peer bb91e016fd844be18687062ddf7ff629: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:57.120275 1329 raft_consensus.cc:1261] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0 [term 1 FOLLOWER]: Refusing update from remote peer bb91e016fd844be18687062ddf7ff629: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:57.121517 1664 consensus_queue.cc:1035] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b735604244554b9991b4925782b8e0f0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:57.122000 1661 consensus_queue.cc:1035] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f60545f5654e68937796bb3ba7e238" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:57.150354 1187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:57.150686 1187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:57.154800 1187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38cbb397218c4e6893eb57c99daff214: Sending DeleteTablet for 3 replicas of tablet b9034f12ce564113bb6b14955d996d53
I20260923 17:21:57.156214 1309 tablet_service.cc:1514] Processing DeleteTablet for tablet b9034f12ce564113bb6b14955d996d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:57 UTC) from {username='slave'} at 127.0.0.1:43304
I20260923 17:21:57.156209 1440 tablet_service.cc:1514] Processing DeleteTablet for tablet b9034f12ce564113bb6b14955d996d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:57 UTC) from {username='slave'} at 127.0.0.1:58238
I20260923 17:21:57.156217 1570 tablet_service.cc:1514] Processing DeleteTablet for tablet b9034f12ce564113bb6b14955d996d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:57 UTC) from {username='slave'} at 127.0.0.1:53724
I20260923 17:21:57.159646 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 1247
I20260923 17:21:57.171713 1677 tablet_replica.cc:330] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629: stopping tablet replica
I20260923 17:21:57.172250 1677 raft_consensus.cc:2229] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:21:57.172766 1650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.162116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.28.72.193:43529)
0923 17:21:57.162690 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:57.162703 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:57.169319 (+ 6616us) server_negotiation.cc:407] Connection header received
0923 17:21:57.169538 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:57.169548 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:57.169708 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:57.169867 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:57.171639 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:57.171651 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:57.171660 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:57.171694 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:57.171752 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:57.171778 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:57.171904 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:57.172230 (+ 326us) server_negotiation.cc:299] Negotiation successful
0923 17:21:57.172320 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260923 17:21:57.173048 1677 raft_consensus.cc:2258] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:57.174116 1369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:57.175916 1677 ts_tablet_manager.cc:1905] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:57.183342 1678 tablet_replica.cc:330] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238: stopping tablet replica
I20260923 17:21:57.184194 1678 raft_consensus.cc:2229] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:57.184921 1678 raft_consensus.cc:2258] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:57.185106 1677 ts_tablet_manager.cc:1918] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:21:57.185344 1677 log.cc:1196] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/b9034f12ce564113bb6b14955d996d53
I20260923 17:21:57.186775 1677 ts_tablet_manager.cc:1939] T b9034f12ce564113bb6b14955d996d53 P bb91e016fd844be18687062ddf7ff629: Deleting consensus metadata
I20260923 17:21:57.188057 1678 ts_tablet_manager.cc:1905] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:57.188796 1172 catalog_manager.cc:4841] TS bb91e016fd844be18687062ddf7ff629 (127.28.72.194:34109): tablet b9034f12ce564113bb6b14955d996d53 (table test-table [id=7c6ab53e39e54ceb9b83323fe25cd1a8]) successfully deleted
I20260923 17:21:57.199503 1678 ts_tablet_manager.cc:1918] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:21:57.199772 1678 log.cc:1196] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/b9034f12ce564113bb6b14955d996d53
I20260923 17:21:57.200520 1678 ts_tablet_manager.cc:1939] T b9034f12ce564113bb6b14955d996d53 P 02f60545f5654e68937796bb3ba7e238: Deleting consensus metadata
I20260923 17:21:57.203151 1172 catalog_manager.cc:4841] TS 02f60545f5654e68937796bb3ba7e238 (127.28.72.195:33629): tablet b9034f12ce564113bb6b14955d996d53 (table test-table [id=7c6ab53e39e54ceb9b83323fe25cd1a8]) successfully deleted
I20260923 17:21:57.298730 1679 tablet_replica.cc:330] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0: stopping tablet replica
I20260923 17:21:57.299243 1679 raft_consensus.cc:2229] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:57.299748 1679 raft_consensus.cc:2258] T b9034f12ce564113bb6b14955d996d53 P b735604244554b9991b4925782b8e0f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:57.300233 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 1247
I20260923 17:21:57.323899 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 1378
W20260923 17:21:57.323951 1175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.72.193:43529 (error 108)
I20260923 17:21:57.328123 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.324710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.28.72.194:34109)
0923 17:21:57.325262 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:57.325267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:57.325284 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:21:57.325359 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:57.325365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:57.325419 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:57.325508 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:57.326802 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:57.326812 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:57.326820 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:57.326847 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:57.326894 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:57.326917 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:57.327067 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:57.327623 (+ 556us) server_negotiation.cc:299] Negotiation successful
0923 17:21:57.327702 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440}
I20260923 17:21:57.329550 1500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:57.331414 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.330840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:43529 (local address 127.0.0.1:43330)
0923 17:21:57.331012 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.331189 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:43529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 17:21:57.331653 1175 catalog_manager.cc:4584] TS b735604244554b9991b4925782b8e0f0 (127.28.72.193:43529): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9034f12ce564113bb6b14955d996d53: Network error: Client connection negotiation failed: client connection to 127.28.72.193:43529: connect: Connection refused (error 111)
I20260923 17:21:57.385617 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.385057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:43529 (local address 127.0.0.1:43334)
0923 17:21:57.385308 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.385422 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:43529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260923 17:21:57.386829 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.386434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:43529 (local address 127.0.0.1:43336)
0923 17:21:57.386587 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.386686 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:43529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 17:21:57.447377 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.446849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:43529 (local address 127.0.0.1:43352)
0923 17:21:57.447063 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.447209 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:43529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 17:21:57.447439 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 1378
I20260923 17:21:57.452342 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.451907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:43529 (local address 127.0.0.1:43354)
0923 17:21:57.452084 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.452177 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:43529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 17:21:57.466609 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 1508
I20260923 17:21:57.469791 1652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.467470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.28.72.195:33629)
0923 17:21:57.467683 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:57.467690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:57.467762 (+ 72us) server_negotiation.cc:407] Connection header received
0923 17:21:57.468010 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:57.468016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:57.468111 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:57.468217 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:57.469115 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:57.469121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:57.469125 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:57.469143 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:57.469181 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:57.469193 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:57.469322 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:57.469427 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 17:21:57.469489 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:21:57.470386 1630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:57.533499 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.532996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:43529 (local address 127.0.0.1:43370)
0923 17:21:57.533218 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.533323 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:43529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 17:21:57.534977 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.534366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:43529 (local address 127.0.0.1:43372)
0923 17:21:57.534715 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.534816 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:43529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252}
I20260923 17:21:57.559017 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 1508
I20260923 17:21:57.575697 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 1157
I20260923 17:21:57.576828 1218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:57.682971 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 1157
2026-09-23T17:21:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2878 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 17:21:57.726146 28963 mini_oidc.cc:152] Starting JWKS server
I20260923 17:21:57.728861 28963 webserver.cc:460] Webserver started at https://127.0.0.1:45481/ using document root <none> and password file <none>
I20260923 17:21:57.729171 28963 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:21:57.729873 28963 webserver.cc:460] Webserver started at http://127.0.0.1:46349/ using document root <none> and password file <none>
2026-09-23T17:21:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:57Z Disabled control of system clock
I20260923 17:21:57.758970 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:35535
--webserver_interface=127.28.72.254
--webserver_port=0
--builtin_ntp_servers=127.28.72.212:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:35535
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45481/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/testchainca.cert with env {}
W20260923 17:21:58.100653 1691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:58.101008 1691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:58.101099 1691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:58.108896 1691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:58.109035 1691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:58.109072 1691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:58.109123 1691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:58.109169 1691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:21:58.109231 1691 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:21:58.109272 1691 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45481/jwks.json
I20260923 17:21:58.119789 1691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:35535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:35535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45481/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:58.121186 1691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:58.122505 1691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:58.135447 1696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:58.135244 1697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:58.135849 1691 server_base.cc:1034] running on GCE node
W20260923 17:21:58.135269 1699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:58.136879 1691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:58.138334 1691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:58.139545 1691 hybrid_clock.cc:648] HybridClock initialized: now 1790184118139494 us; error 36 us; skew 500 ppm
I20260923 17:21:58.139915 1691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:21:58.142452 1691 webserver.cc:460] Webserver started at http://127.28.72.254:33011/ using document root <none> and password file <none>
I20260923 17:21:58.143267 1691 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:58.143401 1691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:58.143731 1691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:58.146803 1691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "ff5bfe73449f4a8aae4b8988d5d5a35f"
format_stamp: "Formatted at 2026-09-23 17:21:58 on dist-test-slave-j0m4"
server_key: "76b4357ace5aa4f6c8d5219fcc6b7a66"
server_key_iv: "1adbca658391d9dc118ea82079eaad8c"
server_key_version: "encryptionkey@0"
I20260923 17:21:58.147754 1691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "ff5bfe73449f4a8aae4b8988d5d5a35f"
format_stamp: "Formatted at 2026-09-23 17:21:58 on dist-test-slave-j0m4"
server_key: "76b4357ace5aa4f6c8d5219fcc6b7a66"
server_key_iv: "1adbca658391d9dc118ea82079eaad8c"
server_key_version: "encryptionkey@0"
I20260923 17:21:58.153467 1691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:21:58.157337 1705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:58.158986 1691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260923 17:21:58.159225 1691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "ff5bfe73449f4a8aae4b8988d5d5a35f"
format_stamp: "Formatted at 2026-09-23 17:21:58 on dist-test-slave-j0m4"
server_key: "76b4357ace5aa4f6c8d5219fcc6b7a66"
server_key_iv: "1adbca658391d9dc118ea82079eaad8c"
server_key_version: "encryptionkey@0"
I20260923 17:21:58.159492 1691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:58.188735 1691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:58.189715 1691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:58.190081 1691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:58.210748 1691 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:35535
I20260923 17:21:58.210769 1756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:35535 every 8 connection(s)
I20260923 17:21:58.212284 1691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:58.216346 1757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:58.219614 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 1691
I20260923 17:21:58.219811 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:21:58.220600 28963 external_mini_cluster.cc:1428] Setting key 5c9e1f50e4708edce2ff0bb5e641504c
I20260923 17:21:58.230063 1757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f: Bootstrap starting.
I20260923 17:21:58.234074 1757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:58.235482 1757 log.cc:824] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:58.236418 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:58.222841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44920 (local address 127.28.72.254:35535)
0923 17:21:58.224018 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:58.224031 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:58.224077 (+ 46us) server_negotiation.cc:407] Connection header received
0923 17:21:58.225896 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:58.225923 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:58.226488 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:58.226898 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:58.227560 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:58.229087 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:58.231041 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:58.231860 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:58.232040 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:58.233373 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:58.233381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:58.233398 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:58.233965 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:58.234089 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:58.234111 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:58.234290 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:58.234735 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 17:21:58.235064 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":451,"threads_started":1}
I20260923 17:21:58.238600 1757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f: No bootstrap required, opened a new log
I20260923 17:21:58.240525 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:21:58.247648 1757 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5bfe73449f4a8aae4b8988d5d5a35f" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35535 } }
I20260923 17:21:58.248052 1757 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:58.248132 1757 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5bfe73449f4a8aae4b8988d5d5a35f, State: Initialized, Role: FOLLOWER
I20260923 17:21:58.248694 1757 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5bfe73449f4a8aae4b8988d5d5a35f" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35535 } }
I20260923 17:21:58.248915 1757 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:58.249012 1757 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:58.249135 1757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:58.251418 1757 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5bfe73449f4a8aae4b8988d5d5a35f" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35535 } }
I20260923 17:21:58.252014 1757 leader_election.cc:302] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff5bfe73449f4a8aae4b8988d5d5a35f; no voters:
I20260923 17:21:58.252758 1757 leader_election.cc:288] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:58.252871 1762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:58.253688 1762 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [term 1 LEADER]: Becoming Leader. State: Replica: ff5bfe73449f4a8aae4b8988d5d5a35f, State: Running, Role: LEADER
I20260923 17:21:58.254259 1762 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5bfe73449f4a8aae4b8988d5d5a35f" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35535 } }
I20260923 17:21:58.255095 1757 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:58.257544 1763 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff5bfe73449f4a8aae4b8988d5d5a35f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5bfe73449f4a8aae4b8988d5d5a35f" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35535 } } }
I20260923 17:21:58.257987 1763 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:58.257758 1764 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff5bfe73449f4a8aae4b8988d5d5a35f. Latest consensus state: current_term: 1 leader_uuid: "ff5bfe73449f4a8aae4b8988d5d5a35f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5bfe73449f4a8aae4b8988d5d5a35f" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35535 } } }
I20260923 17:21:58.258239 1764 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:58.260784 1769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:58.267026 1769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:58.276911 1769 catalog_manager.cc:1347] Generated new cluster ID: cc8f6512006a4ced83770e9170bed45e
I20260923 17:21:58.277077 1769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:58.284929 1769 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:58.286396 1769 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:58.293850 1769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff5bfe73449f4a8aae4b8988d5d5a35f: Generated new TSK 0
I20260923 17:21:58.294837 1769 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 17:21:58.344233 1787 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 17:21:58.344435 1787 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 17:21:58.345342 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:58.315395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44930 (local address 127.28.72.254:35535)
0923 17:21:58.315608 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:58.315614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:58.316318 (+ 704us) server_negotiation.cc:407] Connection header received
0923 17:21:58.316710 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:58.316714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:58.316792 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:58.316958 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:21:58.317677 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:58.318503 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:58.320760 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:58.321249 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:58.321366 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:58.322099 (+ 733us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:21:58.344772 (+ 22673us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45481/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 17:21:58.345120 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44930: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45481/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260923 17:21:58.345489 1760 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44930: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45481/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 17:21:58.347352 1786 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:21:58.315510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35535 (local address 127.0.0.1:44930)
0923 17:21:58.316188 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:58.316209 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:58.316423 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:58.317024 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:58.317029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:58.317043 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:21:58.317461 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:58.317474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:58.318655 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:58.318663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:58.320555 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:58.320573 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:58.321445 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:58.321451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:58.321644 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:58.321739 (+ 95us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:21:58.345108 (+ 23369us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45481/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 17:21:58.345476 (+ 368us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.72.254:35535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45481/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":286,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.72.254:35535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45481/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 17:21:58.348979 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 1691
I20260923 17:21:58.350287 1752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:58.442293 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 1691
2026-09-23T17:21:58Z chronyd exiting
I20260923 17:21:58.490586 28963 test_util.cc:175] -----------------------------------------------
I20260923 17:21:58.490756 28963 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184070511153-28963-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (769 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 17:21:58.494760 28963 mini_oidc.cc:152] Starting JWKS server
I20260923 17:21:58.496922 28963 webserver.cc:460] Webserver started at https://127.0.0.1:38119/ using document root <none> and password file <none>
I20260923 17:21:58.497258 28963 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:21:58.497946 28963 webserver.cc:460] Webserver started at http://127.0.0.1:44393/ using document root <none> and password file <none>
2026-09-23T17:21:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:58Z Disabled control of system clock
I20260923 17:21:58.526840 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:41473
--webserver_interface=127.28.72.254
--webserver_port=0
--builtin_ntp_servers=127.28.72.212:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:41473
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38119/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/testchainca.cert with env {}
W20260923 17:21:58.873551 1794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:58.873960 1794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:58.874121 1794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:58.881824 1794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:58.882004 1794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:58.882097 1794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:58.882203 1794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:58.882287 1794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:21:58.882484 1794 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:21:58.882561 1794 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38119/jwks.json
I20260923 17:21:58.893080 1794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:41473
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:41473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38119/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:58.894552 1794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:58.895944 1794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:58.908274 1799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:58.908243 1802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:58.908195 1800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:58.909410 1794 server_base.cc:1034] running on GCE node
I20260923 17:21:58.910198 1794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:58.911823 1794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:58.913051 1794 hybrid_clock.cc:648] HybridClock initialized: now 1790184118912997 us; error 47 us; skew 500 ppm
I20260923 17:21:58.913573 1794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:21:58.921656 1794 webserver.cc:460] Webserver started at http://127.28.72.254:38593/ using document root <none> and password file <none>
I20260923 17:21:58.922493 1794 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:58.922624 1794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:58.922979 1794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:58.926054 1794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "0d3c3c5d3df740c5802d42c02bd19bbf"
format_stamp: "Formatted at 2026-09-23 17:21:58 on dist-test-slave-j0m4"
server_key: "c44be1b046aba9dfc885dea95659e4f1"
server_key_iv: "0d342c700b66732ba5581867044812c2"
server_key_version: "encryptionkey@0"
I20260923 17:21:58.926895 1794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "0d3c3c5d3df740c5802d42c02bd19bbf"
format_stamp: "Formatted at 2026-09-23 17:21:58 on dist-test-slave-j0m4"
server_key: "c44be1b046aba9dfc885dea95659e4f1"
server_key_iv: "0d342c700b66732ba5581867044812c2"
server_key_version: "encryptionkey@0"
I20260923 17:21:58.932430 1794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:21:58.936262 1808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:58.938048 1794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:21:58.938264 1794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "0d3c3c5d3df740c5802d42c02bd19bbf"
format_stamp: "Formatted at 2026-09-23 17:21:58 on dist-test-slave-j0m4"
server_key: "c44be1b046aba9dfc885dea95659e4f1"
server_key_iv: "0d342c700b66732ba5581867044812c2"
server_key_version: "encryptionkey@0"
I20260923 17:21:58.938516 1794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:58.980971 1794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:58.981900 1794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:58.982229 1794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:59.003559 1794 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:41473
I20260923 17:21:59.003670 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:41473 every 8 connection(s)
I20260923 17:21:59.005266 1794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:59.009187 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 1794
I20260923 17:21:59.009400 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:21:59.009578 1860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:59.010118 28963 external_mini_cluster.cc:1428] Setting key ee61cb9a6c8183f5e2aff4837c73cedb
I20260923 17:21:59.024724 1860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf: Bootstrap starting.
I20260923 17:21:59.026669 1863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:59.012081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60962 (local address 127.28.72.254:41473)
0923 17:21:59.013215 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:59.013234 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:59.013278 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:21:59.015880 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:59.015909 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:59.016538 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:59.016932 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:59.017623 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.019006 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.021293 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.022288 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.022477 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:59.023945 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:59.023954 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:59.023971 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:59.024707 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:59.024807 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:59.024835 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:59.024997 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:59.025309 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 17:21:59.025656 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":431,"threads_started":1}
I20260923 17:21:59.028469 1860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:59.029808 1860 log.cc:824] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:59.030951 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:21:59.033166 1860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf: No bootstrap required, opened a new log
I20260923 17:21:59.042057 1860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3c3c5d3df740c5802d42c02bd19bbf" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 41473 } }
I20260923 17:21:59.042610 1860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:59.042760 1860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3c3c5d3df740c5802d42c02bd19bbf, State: Initialized, Role: FOLLOWER
I20260923 17:21:59.043623 1860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3c3c5d3df740c5802d42c02bd19bbf" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 41473 } }
I20260923 17:21:59.043956 1860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:59.044148 1860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:59.044443 1860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:59.047079 1860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3c3c5d3df740c5802d42c02bd19bbf" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 41473 } }
I20260923 17:21:59.047757 1860 leader_election.cc:302] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d3c3c5d3df740c5802d42c02bd19bbf; no voters:
I20260923 17:21:59.048458 1860 leader_election.cc:288] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:59.048657 1865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:59.049464 1865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [term 1 LEADER]: Becoming Leader. State: Replica: 0d3c3c5d3df740c5802d42c02bd19bbf, State: Running, Role: LEADER
I20260923 17:21:59.050027 1865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3c3c5d3df740c5802d42c02bd19bbf" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 41473 } }
I20260923 17:21:59.050799 1860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:59.053149 1866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d3c3c5d3df740c5802d42c02bd19bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3c3c5d3df740c5802d42c02bd19bbf" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 41473 } } }
I20260923 17:21:59.053819 1866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:59.053771 1867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3c3c5d3df740c5802d42c02bd19bbf. Latest consensus state: current_term: 1 leader_uuid: "0d3c3c5d3df740c5802d42c02bd19bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3c3c5d3df740c5802d42c02bd19bbf" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 41473 } } }
I20260923 17:21:59.054788 1867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:59.055856 1871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:59.061890 1871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:59.079380 1871 catalog_manager.cc:1347] Generated new cluster ID: 0b8fff037dfc4ceba6f459a2bbbd5077
I20260923 17:21:59.079615 1871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:59.115841 1871 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:59.117287 1871 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:59.132335 1871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d3c3c5d3df740c5802d42c02bd19bbf: Generated new TSK 0
I20260923 17:21:59.134738 1871 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 17:21:59.181752 1890 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 17:21:59.181907 1890 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 17:21:59.183301 1863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:59.161008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60974 (local address 127.28.72.254:41473)
0923 17:21:59.161203 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:59.161210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:59.161800 (+ 590us) server_negotiation.cc:407] Connection header received
0923 17:21:59.162037 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:59.162043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:59.162162 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:59.162354 (+ 192us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:21:59.162876 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.164074 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.165693 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.166690 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.166875 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:59.167613 (+ 738us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:21:59.182519 (+ 14906us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 17:21:59.182935 (+ 416us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60974: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260923 17:21:59.183434 1863 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60974: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260923 17:21:59.183502 1889 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:21:59.160984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:41473 (local address 127.0.0.1:60974)
0923 17:21:59.161690 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:59.161707 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:59.161840 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:59.162380 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:59.162384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:59.162394 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:21:59.162715 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:59.162723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.164302 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.164306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:59.165501 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:59.165515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.166950 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.166957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:59.167230 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:59.167282 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:21:59.182793 (+ 15511us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 17:21:59.183029 (+ 236us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.72.254:41473: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":382,"threads_started":1}
I20260923 17:21:59.184785 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 1794
I20260923 17:21:59.186048 1855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:21:59.285084 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 1794
2026-09-23T17:21:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (847 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 17:21:59.342535 28963 mini_oidc.cc:152] Starting JWKS server
I20260923 17:21:59.344605 28963 webserver.cc:460] Webserver started at https://127.0.0.1:42505/ using document root <none> and password file <none>
I20260923 17:21:59.344864 28963 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:21:59.345652 28963 webserver.cc:460] Webserver started at http://127.0.0.1:46775/ using document root <none> and password file <none>
2026-09-23T17:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:59Z Disabled control of system clock
I20260923 17:21:59.378165 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:46639
--webserver_interface=127.28.72.254
--webserver_port=0
--builtin_ntp_servers=127.28.72.212:46069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:46639
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42505/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 17:21:59.730783 1897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:59.731236 1897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:59.731401 1897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:59.739451 1897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:59.739639 1897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:59.739729 1897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:59.739835 1897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:59.739923 1897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:21:59.740011 1897 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:21:59.740082 1897 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42505/jwks.json
I20260923 17:21:59.750169 1897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:46069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:46639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:46639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42505/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:21:59.751612 1897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:59.753608 1897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:59.766419 1905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:59.766621 1902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:59.766765 1903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:59.767508 1897 server_base.cc:1034] running on GCE node
I20260923 17:21:59.768429 1897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:59.770216 1897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:59.771557 1897 hybrid_clock.cc:648] HybridClock initialized: now 1790184119771522 us; error 55 us; skew 500 ppm
I20260923 17:21:59.772022 1897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:21:59.774940 1897 webserver.cc:460] Webserver started at http://127.28.72.254:45907/ using document root <none> and password file <none>
I20260923 17:21:59.775902 1897 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:59.776057 1897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:59.776433 1897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:59.780015 1897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "8ab79f2f33874379b1dd493f36d362e8"
format_stamp: "Formatted at 2026-09-23 17:21:59 on dist-test-slave-j0m4"
server_key: "065f811f714c5caa607e1ee50302de91"
server_key_iv: "e7eef3ccb89c49ff191693ead924a858"
server_key_version: "encryptionkey@0"
I20260923 17:21:59.780946 1897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "8ab79f2f33874379b1dd493f36d362e8"
format_stamp: "Formatted at 2026-09-23 17:21:59 on dist-test-slave-j0m4"
server_key: "065f811f714c5caa607e1ee50302de91"
server_key_iv: "e7eef3ccb89c49ff191693ead924a858"
server_key_version: "encryptionkey@0"
I20260923 17:21:59.786808 1897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 17:21:59.790969 1911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:59.793047 1897 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:21:59.793376 1897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "8ab79f2f33874379b1dd493f36d362e8"
format_stamp: "Formatted at 2026-09-23 17:21:59 on dist-test-slave-j0m4"
server_key: "065f811f714c5caa607e1ee50302de91"
server_key_iv: "e7eef3ccb89c49ff191693ead924a858"
server_key_version: "encryptionkey@0"
I20260923 17:21:59.793665 1897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:59.811738 1897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:59.812728 1897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:59.813079 1897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:59.834950 1897 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:46639
I20260923 17:21:59.834972 1962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:46639 every 8 connection(s)
I20260923 17:21:59.836561 1897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:59.840588 1963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:59.841353 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 1897
I20260923 17:21:59.841566 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:21:59.842422 28963 external_mini_cluster.cc:1428] Setting key 2c75ab355b6676804a5434cf2928f4bb
I20260923 17:21:59.856492 1963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8: Bootstrap starting.
I20260923 17:21:59.858109 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:59.844589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.28.72.254:46639)
0923 17:21:59.845699 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:59.845712 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:59.845756 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:21:59.847692 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:59.847719 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:59.848275 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:59.848647 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:59.849364 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.850716 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.852941 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.853727 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.853888 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:59.855380 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:59.855388 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:59.855406 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:21:59.856047 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:59.856190 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:21:59.856241 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:59.856345 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:59.856653 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 17:21:59.857034 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":388,"threads_started":1}
I20260923 17:21:59.860152 1963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:59.861515 1963 log.cc:824] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:59.862205 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:21:59.864998 1963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8: No bootstrap required, opened a new log
I20260923 17:21:59.873800 1963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab79f2f33874379b1dd493f36d362e8" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46639 } }
I20260923 17:21:59.874310 1963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:59.874439 1963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab79f2f33874379b1dd493f36d362e8, State: Initialized, Role: FOLLOWER
I20260923 17:21:59.875213 1963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab79f2f33874379b1dd493f36d362e8" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46639 } }
I20260923 17:21:59.875495 1963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:59.875648 1963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:59.875829 1963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:59.878337 1963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab79f2f33874379b1dd493f36d362e8" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46639 } }
I20260923 17:21:59.878928 1963 leader_election.cc:302] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ab79f2f33874379b1dd493f36d362e8; no voters:
I20260923 17:21:59.879649 1963 leader_election.cc:288] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:59.879841 1968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:59.880642 1968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [term 1 LEADER]: Becoming Leader. State: Replica: 8ab79f2f33874379b1dd493f36d362e8, State: Running, Role: LEADER
I20260923 17:21:59.881278 1968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab79f2f33874379b1dd493f36d362e8" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46639 } }
I20260923 17:21:59.882017 1963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:59.884219 1969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ab79f2f33874379b1dd493f36d362e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab79f2f33874379b1dd493f36d362e8" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46639 } } }
I20260923 17:21:59.884599 1969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:59.885701 1970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ab79f2f33874379b1dd493f36d362e8. Latest consensus state: current_term: 1 leader_uuid: "8ab79f2f33874379b1dd493f36d362e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab79f2f33874379b1dd493f36d362e8" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46639 } } }
I20260923 17:21:59.886785 1970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8 [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:59.887853 1974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:59.896018 1974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:59.913394 1974 catalog_manager.cc:1347] Generated new cluster ID: f8ec144fd75642418d053e9429757059
I20260923 17:21:59.913539 1974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:59.936853 1974 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:59.938337 1974 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:59.951985 1974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ab79f2f33874379b1dd493f36d362e8: Generated new TSK 0
I20260923 17:21:59.953224 1974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:21:59.993710 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:59.984886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.28.72.254:46639)
0923 17:21:59.985104 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:59.985110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:59.985433 (+ 323us) server_negotiation.cc:407] Connection header received
0923 17:21:59.985661 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:59.985667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:59.985769 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:59.985957 (+ 188us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:21:59.986428 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.987661 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.989151 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.989835 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.989982 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:59.990562 (+ 580us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:21:59.993041 (+ 2479us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 17:21:59.993393 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60620: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":85}
W20260923 17:21:59.993844 1966 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60620: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 17:21:59.995592 1992 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:21:59.984716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46639 (local address 127.0.0.1:60620)
0923 17:21:59.985325 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:59.985341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:59.985481 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:59.985976 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:59.985979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:59.985989 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:21:59.986260 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:59.986268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.987831 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.987837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:59.988995 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:59.989010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.990048 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.990053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:59.990200 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:59.990237 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:21:59.993261 (+ 3024us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 17:21:59.995134 (+ 1873us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.72.254:46639: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":1529,"thread_start_us":328,"threads_started":1}
I20260923 17:21:59.997627 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 1897
I20260923 17:21:59.998978 1958 generic_service.cc:196] Checking for leaks (request via RPC)
E20260923 17:22:00.008380 1993 webserver.cc:533] Webserver: sslize failed:
E20260923 17:22:00.008554 1993 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 17:22:00.094072 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 1897
2026-09-23T17:22:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (808 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T17:22:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:22:00Z Disabled control of system clock
I20260923 17:22:00.180122 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:46437
--webserver_interface=127.28.72.254
--webserver_port=0
--builtin_ntp_servers=127.28.72.212:46467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:46437
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/testchainca.cert with env {}
W20260923 17:22:00.518492 1998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:00.518831 1998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:00.518944 1998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:00.526455 1998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:00.526593 1998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:00.526631 1998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:00.526683 1998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:00.526728 1998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:22:00.526783 1998 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:22:00.526823 1998 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 17:22:00.537034 1998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:46467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:46437
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:46437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:00.538378 1998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:00.539747 1998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:00.552802 2004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:00.552894 2003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:00.553161 2006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:00.553753 1998 server_base.cc:1034] running on GCE node
I20260923 17:22:00.554661 1998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:00.556298 1998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:00.557556 1998 hybrid_clock.cc:648] HybridClock initialized: now 1790184120557493 us; error 61 us; skew 500 ppm
I20260923 17:22:00.557940 1998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:22:00.560688 1998 webserver.cc:460] Webserver started at http://127.28.72.254:34485/ using document root <none> and password file <none>
I20260923 17:22:00.561430 1998 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:00.561561 1998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:00.561869 1998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:00.564911 1998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "d3c0be76c7c6404abbd37b649154c569"
format_stamp: "Formatted at 2026-09-23 17:22:00 on dist-test-slave-j0m4"
server_key: "cafcf045c9bdec25dbd92ebe199a829d"
server_key_iv: "be0066aa9b76e59044d70f8fd9c41d6c"
server_key_version: "encryptionkey@0"
I20260923 17:22:00.565781 1998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "d3c0be76c7c6404abbd37b649154c569"
format_stamp: "Formatted at 2026-09-23 17:22:00 on dist-test-slave-j0m4"
server_key: "cafcf045c9bdec25dbd92ebe199a829d"
server_key_iv: "be0066aa9b76e59044d70f8fd9c41d6c"
server_key_version: "encryptionkey@0"
I20260923 17:22:00.571923 1998 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260923 17:22:00.576264 2012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:00.578016 1998 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20260923 17:22:00.578243 1998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "d3c0be76c7c6404abbd37b649154c569"
format_stamp: "Formatted at 2026-09-23 17:22:00 on dist-test-slave-j0m4"
server_key: "cafcf045c9bdec25dbd92ebe199a829d"
server_key_iv: "be0066aa9b76e59044d70f8fd9c41d6c"
server_key_version: "encryptionkey@0"
I20260923 17:22:00.578510 1998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:00.598717 1998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:00.599659 1998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:00.599972 1998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:00.620187 1998 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:46437
I20260923 17:22:00.620229 2063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:46437 every 8 connection(s)
I20260923 17:22:00.621639 1998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:00.622449 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 1998
I20260923 17:22:00.622669 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:22:00.623468 28963 external_mini_cluster.cc:1428] Setting key e0d6da6fe397c60ff1f3049433b0a8b7
I20260923 17:22:00.627008 2064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:00.637806 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.625356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.28.72.254:46437)
0923 17:22:00.626517 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.626537 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.626592 (+ 55us) server_negotiation.cc:407] Connection header received
0923 17:22:00.628413 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.628435 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.628884 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.629264 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.629893 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.631293 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.632638 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.633350 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.633570 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.635096 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.635105 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.635174 (+ 69us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:00.635843 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.635933 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:00.635960 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.636136 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.636397 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.636752 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":362,"threads_started":1}
I20260923 17:22:00.638857 2064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569: Bootstrap starting.
I20260923 17:22:00.642584 2064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:00.642606 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:22:00.643705 2064 log.cc:824] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:00.647271 2064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569: No bootstrap required, opened a new log
I20260923 17:22:00.655069 2064 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c0be76c7c6404abbd37b649154c569" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46437 } }
I20260923 17:22:00.655704 2064 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:00.655864 2064 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c0be76c7c6404abbd37b649154c569, State: Initialized, Role: FOLLOWER
I20260923 17:22:00.656745 2064 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c0be76c7c6404abbd37b649154c569" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46437 } }
I20260923 17:22:00.657055 2064 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:00.657218 2064 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:00.657392 2064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:00.659920 2064 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c0be76c7c6404abbd37b649154c569" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46437 } }
I20260923 17:22:00.660533 2064 leader_election.cc:302] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c0be76c7c6404abbd37b649154c569; no voters:
I20260923 17:22:00.661262 2064 leader_election.cc:288] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:22:00.661458 2069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:00.662909 2069 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [term 1 LEADER]: Becoming Leader. State: Replica: d3c0be76c7c6404abbd37b649154c569, State: Running, Role: LEADER
I20260923 17:22:00.663944 2064 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:00.663985 2069 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c0be76c7c6404abbd37b649154c569" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46437 } }
I20260923 17:22:00.667788 2071 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c0be76c7c6404abbd37b649154c569. Latest consensus state: current_term: 1 leader_uuid: "d3c0be76c7c6404abbd37b649154c569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c0be76c7c6404abbd37b649154c569" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46437 } } }
I20260923 17:22:00.668347 2071 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:00.669050 2070 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3c0be76c7c6404abbd37b649154c569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c0be76c7c6404abbd37b649154c569" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46437 } } }
I20260923 17:22:00.669570 2070 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:00.670184 2078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:00.676848 2078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:00.688969 2078 catalog_manager.cc:1347] Generated new cluster ID: 94ffe2378bc24345b93d0af3e45c7da5
I20260923 17:22:00.689158 2078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:00.718744 2078 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:22:00.721026 2078 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:00.729207 2078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3c0be76c7c6404abbd37b649154c569: Generated new TSK 0
I20260923 17:22:00.730144 2078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:22:00.916992 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.759625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.28.72.254:46437)
0923 17:22:00.759840 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.759848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.760244 (+ 396us) server_negotiation.cc:407] Connection header received
0923 17:22:00.760467 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.760475 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.760641 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.760830 (+ 189us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:22:00.761328 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.762590 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.764269 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.765099 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.765266 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.765790 (+ 524us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:22:00.916208 (+150418us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 17:22:00.916656 (+ 448us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37758: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 17:22:00.917136 2067 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37758: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 17:22:00.917224 2093 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:00.759462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46437 (local address 127.0.0.1:37758)
0923 17:22:00.760142 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.760159 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:00.760282 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:00.760898 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:00.760902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:00.760913 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:22:00.761165 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.761174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.762767 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.762772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.764117 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.764131 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.765299 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.765303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.765431 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.765467 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:22:00.916563 (+151096us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 17:22:00.916815 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.72.254:46437: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":386,"threads_started":1}
I20260923 17:22:00.918418 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 1998
I20260923 17:22:00.920174 2059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:01.031311 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 1998
2026-09-23T17:22:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (918 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (444 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (449 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:22:02 dist-test-slave-j0m4 krb5kdc[2108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:22:02 dist-test-slave-j0m4 krb5kdc[2108](info): set up 2 sockets
Sep 23 17:22:02 dist-test-slave-j0m4 krb5kdc[2108](info): commencing operation
krb5kdc: starting...
W20260923 17:22:03.052713 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:22:03 dist-test-slave-j0m4 krb5kdc[2108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:22:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:22:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:22:03.284317 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:42377
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:42377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:03.625388 2121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:03.625792 2121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:03.625957 2121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:03.633424 2121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:03.633613 2121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:03.633716 2121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:03.633810 2121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:03.633893 2121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:03.644156 2121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:42377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:42377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:03.645685 2121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:03.647068 2121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:03.659590 2129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:03.659788 2126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:03.659590 2127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:03.660416 2121 server_base.cc:1034] running on GCE node
I20260923 17:22:03.661339 2121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:03.662892 2121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:03.664206 2121 hybrid_clock.cc:648] HybridClock initialized: now 1790184123664157 us; error 81 us; skew 500 ppm
I20260923 17:22:03.664614 2121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:03 dist-test-slave-j0m4 krb5kdc[2108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184123, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:03.668138 2121 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:22:03.669826 2121 webserver.cc:460] Webserver started at http://127.28.72.254:43493/ using document root <none> and password file <none>
I20260923 17:22:03.670583 2121 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:03.670732 2121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:03.671065 2121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:03.674314 2121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "fa9f201ddbd04a2b8414c48db6af7430"
format_stamp: "Formatted at 2026-09-23 17:22:03 on dist-test-slave-j0m4"
server_key: "b5eb8881e4e1b0b3bd9a27b4aa3a0676"
server_key_iv: "47548232a1d1e75638e865a711bcc3dc"
server_key_version: "encryptionkey@0"
I20260923 17:22:03.675242 2121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "fa9f201ddbd04a2b8414c48db6af7430"
format_stamp: "Formatted at 2026-09-23 17:22:03 on dist-test-slave-j0m4"
server_key: "b5eb8881e4e1b0b3bd9a27b4aa3a0676"
server_key_iv: "47548232a1d1e75638e865a711bcc3dc"
server_key_version: "encryptionkey@0"
I20260923 17:22:03.680985 2121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:22:03.684868 2136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:03.686467 2121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260923 17:22:03.686707 2121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "fa9f201ddbd04a2b8414c48db6af7430"
format_stamp: "Formatted at 2026-09-23 17:22:03 on dist-test-slave-j0m4"
server_key: "b5eb8881e4e1b0b3bd9a27b4aa3a0676"
server_key_iv: "47548232a1d1e75638e865a711bcc3dc"
server_key_version: "encryptionkey@0"
I20260923 17:22:03.686954 2121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:03.705322 2121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:03.710412 2121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:03.710747 2121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:03.731091 2121 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:42377
I20260923 17:22:03.731232 2187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:42377 every 8 connection(s)
I20260923 17:22:03.732686 2121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:03.736928 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 2121
I20260923 17:22:03.737213 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:22:03.737263 2188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:03.737929 28963 external_mini_cluster.cc:1428] Setting key 9fc1a2abcecb9a9997b00d9e80102c5c
I20260923 17:22:03.752157 2188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430: Bootstrap starting.
Sep 23 17:22:03 dist-test-slave-j0m4 krb5kdc[2108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:22:03.756285 2188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:03.757387 2188 log.cc:824] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:03.761181 2188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430: No bootstrap required, opened a new log
I20260923 17:22:03.764467 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:03.740226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.28.72.254:42377)
0923 17:22:03.741093 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:03.741105 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:03.741147 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:22:03.743661 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:03.743692 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:03.744186 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:03.744574 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:03.746359 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:03.747861 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:03.749778 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:03.750545 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:03.750686 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:03.755835 (+ 5149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:03.755844 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:03.755861 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:03.755904 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:03.759709 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:03.760717 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:03.760724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:03.760731 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:03.760898 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:03.761431 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:03.761435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:03.761437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:03.762031 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:03.762267 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:03.762791 (+ 524us) server_negotiation.cc:299] Negotiation successful
0923 17:22:03.763333 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":415,"threads_started":1}
I20260923 17:22:03.770064 2188 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9f201ddbd04a2b8414c48db6af7430" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42377 } }
I20260923 17:22:03.770463 2188 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:03.770594 2188 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9f201ddbd04a2b8414c48db6af7430, State: Initialized, Role: FOLLOWER
I20260923 17:22:03.771379 2188 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9f201ddbd04a2b8414c48db6af7430" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42377 } }
I20260923 17:22:03.771603 2188 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:03.771772 2188 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:03.771920 2188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:03.774205 2188 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9f201ddbd04a2b8414c48db6af7430" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42377 } }
I20260923 17:22:03.774832 2188 leader_election.cc:302] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa9f201ddbd04a2b8414c48db6af7430; no voters:
I20260923 17:22:03.775491 2188 leader_election.cc:288] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:22:03.775674 2193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:03.776505 2193 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [term 1 LEADER]: Becoming Leader. State: Replica: fa9f201ddbd04a2b8414c48db6af7430, State: Running, Role: LEADER
I20260923 17:22:03.777086 2193 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9f201ddbd04a2b8414c48db6af7430" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42377 } }
I20260923 17:22:03.777709 2188 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:03.780033 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa9f201ddbd04a2b8414c48db6af7430. Latest consensus state: current_term: 1 leader_uuid: "fa9f201ddbd04a2b8414c48db6af7430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9f201ddbd04a2b8414c48db6af7430" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42377 } } }
I20260923 17:22:03.780503 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:03.781122 2194 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa9f201ddbd04a2b8414c48db6af7430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9f201ddbd04a2b8414c48db6af7430" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42377 } } }
I20260923 17:22:03.781442 2194 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:03.783838 2200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:03.789681 2200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:03.799994 2200 catalog_manager.cc:1347] Generated new cluster ID: 3795cc46eb324b6d87d933fe59486731
I20260923 17:22:03.800134 2200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:03.817296 2200 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:22:03.818784 2200 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:03.828995 2200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa9f201ddbd04a2b8414c48db6af7430: Generated new TSK 0
I20260923 17:22:03.829896 2200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:22:03.896462 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42377
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:04.264568 2215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:04.264882 2215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:04.265092 2215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:04.272890 2215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:04.273025 2215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:04.273149 2215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:22:04.283896 2215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:04.285252 2215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:04.286638 2215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:04.301879 2223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:04.301923 2221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:04.302126 2220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:04.302708 2215 server_base.cc:1034] running on GCE node
I20260923 17:22:04.303521 2215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:04.304596 2215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:04.305857 2215 hybrid_clock.cc:648] HybridClock initialized: now 1790184124305810 us; error 36 us; skew 500 ppm
I20260923 17:22:04.306192 2215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:04 dist-test-slave-j0m4 krb5kdc[2108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184124, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:04.314472 2215 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:22:04.316193 2215 webserver.cc:460] Webserver started at http://127.28.72.193:37549/ using document root <none> and password file <none>
I20260923 17:22:04.316882 2215 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:04.316993 2215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:04.317253 2215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:04.320356 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "90cf7dab41b4480e82233989a679ed86"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-j0m4"
server_key: "800be6e7343bb897862d2908734a5f9d"
server_key_iv: "087e21b0b6bab3c15331cd98e12f234c"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.321117 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "90cf7dab41b4480e82233989a679ed86"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-j0m4"
server_key: "800be6e7343bb897862d2908734a5f9d"
server_key_iv: "087e21b0b6bab3c15331cd98e12f234c"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.326447 2215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:22:04.330119 2230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:04.331655 2215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:04.331813 2215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "90cf7dab41b4480e82233989a679ed86"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-j0m4"
server_key: "800be6e7343bb897862d2908734a5f9d"
server_key_iv: "087e21b0b6bab3c15331cd98e12f234c"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.332001 2215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:04.352412 2215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:04.357218 2215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:04.357492 2215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:04.358464 2215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:04.359972 2215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:04.360082 2215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:04.360174 2215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:04.360221 2215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:04.394124 2215 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:42037
I20260923 17:22:04.394248 2342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:42037 every 8 connection(s)
I20260923 17:22:04.395609 2215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:22:04.403599 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 2215
I20260923 17:22:04.403821 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:22:04.404469 28963 external_mini_cluster.cc:1428] Setting key aa21cccd1e1192bdac070322596075b7
Sep 23 17:22:04 dist-test-slave-j0m4 krb5kdc[2108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184124, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:22:04.415897 2345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:04.398287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:58365 (local address 127.28.72.254:42377)
0923 17:22:04.398754 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:04.398761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:04.399828 (+ 1067us) server_negotiation.cc:407] Connection header received
0923 17:22:04.401655 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:04.401659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:04.401729 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:04.401834 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:04.403474 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.404317 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.405981 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.406377 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.406531 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:04.410061 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:04.410072 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:04.410076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:04.410109 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:04.412724 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:04.413518 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:04.413524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:04.413526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:04.413639 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:04.414076 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:04.414082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:04.414085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:04.414369 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:04.414527 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:04.415294 (+ 767us) server_negotiation.cc:299] Negotiation successful
0923 17:22:04.415531 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":227,"threads_started":1}
I20260923 17:22:04.416910 2346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:04.398896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:42377 (local address 127.28.72.193:58365)
0923 17:22:04.399659 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:04.399699 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:04.401370 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:04.402082 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:04.402090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:04.402610 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:04.403280 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:04.403301 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.404492 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.404496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:04.405822 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:04.405835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.406623 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.406632 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:04.406890 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:04.407643 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:04.407670 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:04.409821 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:04.412885 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:04.412896 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:04.412916 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:04.413372 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:04.413757 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:04.413762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:04.413765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:04.413954 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:04.414497 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:04.414506 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:04.414878 (+ 372us) client_negotiation.cc:769] Sending connection context
0923 17:22:04.415176 (+ 298us) client_negotiation.cc:240] Negotiation successful
0923 17:22:04.415742 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":322,"threads_started":1}
I20260923 17:22:04.418582 2343 heartbeater.cc:347] Connected to a master server at 127.28.72.254:42377
I20260923 17:22:04.418948 2343 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:04.419956 2343 heartbeater.cc:511] Master 127.28.72.254:42377 requested a full tablet report, sending...
I20260923 17:22:04.422726 2153 ts_manager.cc:194] Registered new tserver with Master: 90cf7dab41b4480e82233989a679ed86 (127.28.72.193:42037)
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
I20260923 17:22:04.424947 2153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:58365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:22:04.456176 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42377
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:04.812875 2349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:04.813251 2349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:04.813421 2349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:04.821205 2349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:04.821393 2349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:04.821590 2349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:22:04.832139 2349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:04.833583 2349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:04.835029 2349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:04.849915 2355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:04.849915 2354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:04.849915 2357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:04.850908 2349 server_base.cc:1034] running on GCE node
I20260923 17:22:04.851686 2349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:04.852685 2349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:04.853950 2349 hybrid_clock.cc:648] HybridClock initialized: now 1790184124853899 us; error 41 us; skew 500 ppm
I20260923 17:22:04.854257 2349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:04 dist-test-slave-j0m4 krb5kdc[2108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184124, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:04.857781 2349 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:22:04.859596 2349 webserver.cc:460] Webserver started at http://127.28.72.194:35937/ using document root <none> and password file <none>
I20260923 17:22:04.860298 2349 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:04.860405 2349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:04.860661 2349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:04.863750 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "f08d9b6ffc7f411d99e8d9d7079ed543"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-j0m4"
server_key: "0501ae47413de267d552c1413a35d712"
server_key_iv: "fbf0134b11a8794698300caa22cd00e6"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.864677 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "f08d9b6ffc7f411d99e8d9d7079ed543"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-j0m4"
server_key: "0501ae47413de267d552c1413a35d712"
server_key_iv: "fbf0134b11a8794698300caa22cd00e6"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.870456 2349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 17:22:04.874589 2364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:04.876370 2349 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260923 17:22:04.876538 2349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "f08d9b6ffc7f411d99e8d9d7079ed543"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-j0m4"
server_key: "0501ae47413de267d552c1413a35d712"
server_key_iv: "fbf0134b11a8794698300caa22cd00e6"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.876739 2349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:04.901415 2349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:04.906008 2349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:04.906265 2349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:04.907323 2349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:04.908618 2349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:04.908691 2349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:04.908785 2349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:04.908828 2349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:04.944684 2349 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:34501
I20260923 17:22:04.944797 2476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:34501 every 8 connection(s)
I20260923 17:22:04.946054 2349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:22:04.950776 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 2349
I20260923 17:22:04.951045 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:22:04.951617 28963 external_mini_cluster.cc:1428] Setting key 2f2b846d6b17c84dff78eb6b101ffd38
Sep 23 17:22:04 dist-test-slave-j0m4 krb5kdc[2108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184124, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:22:04.972596 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:04.948927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:50543 (local address 127.28.72.254:42377)
0923 17:22:04.949402 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:04.949407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:04.950645 (+ 1238us) server_negotiation.cc:407] Connection header received
0923 17:22:04.953130 (+ 2485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:04.953134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:04.953199 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:04.953294 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:04.955531 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.956274 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.958951 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.959602 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.959716 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:04.964196 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:04.964206 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:04.964214 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:04.964265 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:04.968149 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:04.969446 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:04.969455 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:04.969461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:04.969635 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:04.970140 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:04.970149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:04.970155 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:04.970602 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:04.970776 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:04.971745 (+ 969us) server_negotiation.cc:299] Negotiation successful
0923 17:22:04.972053 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":215,"threads_started":1}
I20260923 17:22:04.973317 2480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:04.949530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:42377 (local address 127.28.72.194:50543)
0923 17:22:04.950299 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:04.950487 (+ 188us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:04.952770 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:04.953582 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:04.953591 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:04.954332 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:04.955279 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:04.955310 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.956451 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.956458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:04.958715 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:04.958736 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.959995 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.960005 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:04.960387 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:04.961269 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:04.961294 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:04.963877 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:04.968653 (+ 4776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:04.968663 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:04.968680 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:04.969258 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:04.969775 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:04.969779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:04.969781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:04.969990 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:04.970770 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:04.970778 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:04.971279 (+ 501us) client_negotiation.cc:769] Sending connection context
0923 17:22:04.971590 (+ 311us) client_negotiation.cc:240] Negotiation successful
0923 17:22:04.972053 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":305,"threads_started":1}
I20260923 17:22:04.974884 2477 heartbeater.cc:347] Connected to a master server at 127.28.72.254:42377
I20260923 17:22:04.975257 2477 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
I20260923 17:22:04.976053 2477 heartbeater.cc:511] Master 127.28.72.254:42377 requested a full tablet report, sending...
I20260923 17:22:04.977720 2153 ts_manager.cc:194] Registered new tserver with Master: f08d9b6ffc7f411d99e8d9d7079ed543 (127.28.72.194:34501)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
I20260923 17:22:04.978801 2153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:50543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:22:05.011765 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42377
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:05.366236 2483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:05.366622 2483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:05.366806 2483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:05.375023 2483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:05.375224 2483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:05.375414 2483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:22:05.386058 2483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:05.387571 2483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:05.389045 2483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:05.404713 2488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:05.404838 2489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:05.404767 2491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:05.405604 2483 server_base.cc:1034] running on GCE node
I20260923 17:22:05.406283 2483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:05.407351 2483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:05.408604 2483 hybrid_clock.cc:648] HybridClock initialized: now 1790184125408538 us; error 45 us; skew 500 ppm
I20260923 17:22:05.408973 2483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:05 dist-test-slave-j0m4 krb5kdc[2108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:05.412602 2483 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:22:05.414413 2483 webserver.cc:460] Webserver started at http://127.28.72.195:40825/ using document root <none> and password file <none>
I20260923 17:22:05.415269 2483 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:05.415414 2483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:05.415746 2483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:05.419031 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "745a6b204ff24057ac6e5a1c26d06a02"
format_stamp: "Formatted at 2026-09-23 17:22:05 on dist-test-slave-j0m4"
server_key: "b29a8cb9a72d0a3df53d3eb93a7f2d09"
server_key_iv: "8b25e5e3ac2719a087565ccc18b68541"
server_key_version: "encryptionkey@0"
I20260923 17:22:05.420004 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "745a6b204ff24057ac6e5a1c26d06a02"
format_stamp: "Formatted at 2026-09-23 17:22:05 on dist-test-slave-j0m4"
server_key: "b29a8cb9a72d0a3df53d3eb93a7f2d09"
server_key_iv: "8b25e5e3ac2719a087565ccc18b68541"
server_key_version: "encryptionkey@0"
I20260923 17:22:05.425861 2483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:22:05.428602 2343 heartbeater.cc:503] Master 127.28.72.254:42377 was elected leader, sending a full tablet report...
I20260923 17:22:05.429981 2498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:05.431797 2483 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:22:05.431979 2483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "745a6b204ff24057ac6e5a1c26d06a02"
format_stamp: "Formatted at 2026-09-23 17:22:05 on dist-test-slave-j0m4"
server_key: "b29a8cb9a72d0a3df53d3eb93a7f2d09"
server_key_iv: "8b25e5e3ac2719a087565ccc18b68541"
server_key_version: "encryptionkey@0"
I20260923 17:22:05.432173 2483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:05.446645 2483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:05.451408 2483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:05.451715 2483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:05.452725 2483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:05.454174 2483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:05.454247 2483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:05.454411 2483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:05.454475 2483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:05.490094 2483 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:38271
I20260923 17:22:05.490140 2610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:38271 every 8 connection(s)
I20260923 17:22:05.491566 2483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:22:05.494977 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 2483
I20260923 17:22:05.495210 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:22:05.495709 28963 external_mini_cluster.cc:1428] Setting key 98b0a6938d072017df17149310550723
Sep 23 17:22:05 dist-test-slave-j0m4 krb5kdc[2108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:22:05.514479 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.494541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:59935 (local address 127.28.72.254:42377)
0923 17:22:05.495170 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.495180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.496390 (+ 1210us) server_negotiation.cc:407] Connection header received
0923 17:22:05.498688 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.498694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.498794 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.498910 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.500703 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.501721 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.503638 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.504071 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.504183 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.508069 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:05.508075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:05.508079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:05.508113 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:05.511318 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.512207 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.512212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.512214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.512357 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.512774 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.512778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.512780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.513064 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:05.513185 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.513976 (+ 791us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.514164 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":324,"threads_started":1}
I20260923 17:22:05.515632 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.495248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:42377 (local address 127.28.72.195:59935)
0923 17:22:05.496172 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:05.496251 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:05.498350 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:05.499185 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:05.499195 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:05.499848 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:05.500490 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.500515 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.501902 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.501907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.503474 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.503488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.504328 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.504333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.504635 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.505342 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:05.505365 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:05.507787 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:05.511521 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:05.511547 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:05.511568 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:05.512072 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:05.512464 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:05.512467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:05.512470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:05.512659 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:05.513198 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:05.513205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:05.513572 (+ 367us) client_negotiation.cc:769] Sending connection context
0923 17:22:05.513815 (+ 243us) client_negotiation.cc:240] Negotiation successful
0923 17:22:05.514245 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":375,"threads_started":1}
I20260923 17:22:05.517189 2611 heartbeater.cc:347] Connected to a master server at 127.28.72.254:42377
I20260923 17:22:05.517575 2611 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:05.518313 2611 heartbeater.cc:511] Master 127.28.72.254:42377 requested a full tablet report, sending...
I20260923 17:22:05.519896 2153 ts_manager.cc:194] Registered new tserver with Master: 745a6b204ff24057ac6e5a1c26d06a02 (127.28.72.195:38271)
I20260923 17:22:05.520735 2153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:59935
I20260923 17:22:05.523994 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/test-admin.keytab.
Sep 23 17:22:05 dist-test-slave-j0m4 krb5kdc[2108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:05.553961 28963 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 17:22:05.559620 2619 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:05.556843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.28.72.193:42037)
0923 17:22:05.557484 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.557494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.557522 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:22:05.557579 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.557586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.557796 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.557961 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.559011 (+ 1050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49998: BlockingRecv error: recv got EOF from 127.0.0.1:49998 (error 108)
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":334,"threads_started":1}
W20260923 17:22:05.559674 28963 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/krb5cc' of type FILE
Sep 23 17:22:05 dist-test-slave-j0m4 krb5kdc[2108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:05.561506 28963 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:22:05 dist-test-slave-j0m4 krb5kdc[2108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:05.563088 28963 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:22:05 dist-test-slave-j0m4 krb5kdc[2108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:22:05.580565 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.564977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.28.72.193:42037)
0923 17:22:05.565326 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.565333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.565693 (+ 360us) server_negotiation.cc:407] Connection header received
0923 17:22:05.565899 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.565904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.565973 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.566077 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.567138 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.568134 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.569383 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.570024 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.570166 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.574182 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:05.574198 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:05.574214 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:05.574253 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:05.577499 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.578264 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.578278 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.578281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.578411 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.578826 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.578831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.578834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.579299 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:05.579594 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.579950 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.580220 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
W20260923 17:22:05.587049 2619 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:05.585063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.28.72.193:42037)
0923 17:22:05.585219 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.585223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.585587 (+ 364us) server_negotiation.cc:407] Connection header received
0923 17:22:05.585718 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.585721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.585781 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.585863 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.586750 (+ 887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50010: BlockingRecv error: recv got EOF from 127.0.0.1:50010 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260923 17:22:05.587255 28963 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/krb5cc' of type FILE
Sep 23 17:22:05 dist-test-slave-j0m4 krb5kdc[2108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:05.589166 28963 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:22:05 dist-test-slave-j0m4 krb5kdc[2108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:05.590719 28963 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:22:05 dist-test-slave-j0m4 krb5kdc[2108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:22:05.606817 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.592253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.28.72.193:42037)
0923 17:22:05.592509 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.592516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.592832 (+ 316us) server_negotiation.cc:407] Connection header received
0923 17:22:05.592991 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.592995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.593057 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.593144 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.594530 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.595297 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.596926 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.597350 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.597513 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.601550 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:05.601556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:05.601559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:05.601598 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:05.604276 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.604989 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.604993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.604995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.605127 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.605547 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.605551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.605553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.605832 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:05.606024 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.606311 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.606488 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260923 17:22:05.611328 2619 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:05.609439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.28.72.193:42037)
0923 17:22:05.609593 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.609598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.609992 (+ 394us) server_negotiation.cc:407] Connection header received
0923 17:22:05.610108 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.610111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.610178 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.610255 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.611020 (+ 765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50028: BlockingRecv error: recv got EOF from 127.0.0.1:50028 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260923 17:22:05.611511 28963 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184070511153-28963-0/krb5kdc/krb5cc' of type FILE
Sep 23 17:22:05 dist-test-slave-j0m4 krb5kdc[2108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:05.613057 28963 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:22:05 dist-test-slave-j0m4 krb5kdc[2108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:05.614449 28963 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:22:05 dist-test-slave-j0m4 krb5kdc[2108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:22:05.629491 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.616151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.28.72.193:42037)
0923 17:22:05.616360 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.616367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.616519 (+ 152us) server_negotiation.cc:407] Connection header received
0923 17:22:05.616714 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.616724 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.616791 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.616888 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.617826 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.618557 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.619849 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.620282 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.620417 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.624053 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:05.624059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:05.624062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:05.624101 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:05.626797 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.627552 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.627559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.627563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.627718 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.628088 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.628097 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.628099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.628369 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:05.628698 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.628951 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.629170 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:22:05.631228 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 2215
I20260923 17:22:05.644243 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.631967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.28.72.193:42037)
0923 17:22:05.632156 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.632160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.632536 (+ 376us) server_negotiation.cc:407] Connection header received
0923 17:22:05.632645 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.632649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.632708 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.632801 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.633828 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.634609 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.636048 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.636605 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.636703 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.638899 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:05.638904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:05.638907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:05.638940 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:05.641601 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.642296 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.642300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.642303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.642506 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.642831 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.642836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.642838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.643094 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:05.643426 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.643775 (+ 349us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.643963 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:22:05.644985 2338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:05.730146 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 2215
I20260923 17:22:05.750084 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 2349
Sep 23 17:22:05 dist-test-slave-j0m4 krb5kdc[2108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:22:05.766920 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.751045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.28.72.194:34501)
0923 17:22:05.751653 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.751659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.751686 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:22:05.751745 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.751751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.751971 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.752107 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.753172 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.754319 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.755989 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.756592 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.756701 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.760616 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:05.760629 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:05.760647 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:05.760696 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:05.763810 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.764632 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.764640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.764642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.764769 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.765205 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.765209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.765212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.765614 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:05.765860 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.766215 (+ 355us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.766604 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":335,"threads_started":1}
I20260923 17:22:05.768386 2472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:05.852710 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 2349
I20260923 17:22:05.869213 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 2483
Sep 23 17:22:05 dist-test-slave-j0m4 krb5kdc[2108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:22:05.887614 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.870447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.28.72.195:38271)
0923 17:22:05.871253 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.871261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.871296 (+ 35us) server_negotiation.cc:407] Connection header received
0923 17:22:05.871370 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.871377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.871601 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.871768 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.872947 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.874018 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.875485 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.876068 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.876174 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.880753 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:05.880767 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:05.880793 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:05.880831 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:05.884028 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.884955 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.884963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.884965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.885102 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.885555 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.885558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.885560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.885933 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:05.886170 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.886658 (+ 488us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.886992 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":387,"threads_started":1}
I20260923 17:22:05.889353 2606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:05.976781 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 2483
I20260923 17:22:05.993199 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 2121
I20260923 17:22:05.994462 2183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:06.100173 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 2121
2026-09-23T17:22:06Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4178 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:22:06 dist-test-slave-j0m4 krb5kdc[2639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:22:06 dist-test-slave-j0m4 krb5kdc[2639](info): set up 2 sockets
Sep 23 17:22:06 dist-test-slave-j0m4 krb5kdc[2639](info): commencing operation
krb5kdc: starting...
W20260923 17:22:07.248018 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:22:07 dist-test-slave-j0m4 krb5kdc[2639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:22:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:22:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.254.keytab.
Entry for principal oryx/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.254.keytab.
I20260923 17:22:07.483426 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:34367
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.254.keytab
--principal=oryx/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:34367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:07.837965 2652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:07.838359 2652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:07.838538 2652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:07.846114 2652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:07.846306 2652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:07.846421 2652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:07.846529 2652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:07.846613 2652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:07.857414 2652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:34367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.254.keytab
--principal=oryx/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:34367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:07.859026 2652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:07.860459 2652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:07.872876 2658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:07.872903 2660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:07.873138 2657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:07.874698 2652 server_base.cc:1034] running on GCE node
I20260923 17:22:07.875741 2652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:07.877383 2652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:07.878633 2652 hybrid_clock.cc:648] HybridClock initialized: now 1790184127878588 us; error 33 us; skew 500 ppm
I20260923 17:22:07.879098 2652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:07 dist-test-slave-j0m4 krb5kdc[2639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184127, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:07.882738 2652 init.cc:377] Logged in from keytab as oryx/127.28.72.254@KRBTEST.COM (short username oryx)
I20260923 17:22:07.884676 2652 webserver.cc:460] Webserver started at http://127.28.72.254:46559/ using document root <none> and password file <none>
I20260923 17:22:07.885444 2652 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:07.885579 2652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:07.885903 2652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:07.889387 2652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "a9e1d06ed1444e00ae887cb39ee5a144"
format_stamp: "Formatted at 2026-09-23 17:22:07 on dist-test-slave-j0m4"
server_key: "05d1b3e86b26ea7dcd5f03ec6f535079"
server_key_iv: "4f666c6d12e554f43d151f2b8511557a"
server_key_version: "encryptionkey@0"
I20260923 17:22:07.890280 2652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "a9e1d06ed1444e00ae887cb39ee5a144"
format_stamp: "Formatted at 2026-09-23 17:22:07 on dist-test-slave-j0m4"
server_key: "05d1b3e86b26ea7dcd5f03ec6f535079"
server_key_iv: "4f666c6d12e554f43d151f2b8511557a"
server_key_version: "encryptionkey@0"
I20260923 17:22:07.897913 2652 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260923 17:22:07.902457 2667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:07.904388 2652 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:22:07.904630 2652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "a9e1d06ed1444e00ae887cb39ee5a144"
format_stamp: "Formatted at 2026-09-23 17:22:07 on dist-test-slave-j0m4"
server_key: "05d1b3e86b26ea7dcd5f03ec6f535079"
server_key_iv: "4f666c6d12e554f43d151f2b8511557a"
server_key_version: "encryptionkey@0"
I20260923 17:22:07.904868 2652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:07.927527 2652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:07.932312 2652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:07.932634 2652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:07.953020 2652 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:34367
I20260923 17:22:07.953063 2718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:34367 every 8 connection(s)
I20260923 17:22:07.954605 2652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:07.956583 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 2652
I20260923 17:22:07.956841 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:22:07.957511 28963 external_mini_cluster.cc:1428] Setting key 2ffb99c2410cc057e77529c645797a53
I20260923 17:22:07.959174 2719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:22:07 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
I20260923 17:22:07.974076 2719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144: Bootstrap starting.
I20260923 17:22:07.978255 2719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:07.980073 2719 log.cc:824] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:07.983690 2719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144: No bootstrap required, opened a new log
I20260923 17:22:07.984368 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:07.959688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.28.72.254:34367)
0923 17:22:07.960524 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:07.960536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:07.960578 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:22:07.962161 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:07.962185 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:07.962635 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:07.963034 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:07.965156 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:07.966674 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:07.968543 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:07.969346 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:07.969531 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:07.974823 (+ 5292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:07.974834 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:07.974852 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:07.974899 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:07.978364 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:07.979828 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:07.979837 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:07.979849 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:07.980120 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:07.980695 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:07.980704 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:07.980711 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:07.981660 (+ 949us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:07.982018 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:07.982555 (+ 537us) server_negotiation.cc:299] Negotiation successful
0923 17:22:07.983052 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":403,"threads_started":1}
I20260923 17:22:07.992236 2719 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } }
I20260923 17:22:07.992720 2719 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:07.992841 2719 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e1d06ed1444e00ae887cb39ee5a144, State: Initialized, Role: FOLLOWER
I20260923 17:22:07.993501 2719 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } }
I20260923 17:22:07.993769 2719 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:07.993916 2719 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:07.994135 2719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:07.996699 2719 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } }
I20260923 17:22:07.997283 2719 leader_election.cc:302] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9e1d06ed1444e00ae887cb39ee5a144; no voters:
I20260923 17:22:07.997947 2719 leader_election.cc:288] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:22:07.998142 2724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:07.999271 2724 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [term 1 LEADER]: Becoming Leader. State: Replica: a9e1d06ed1444e00ae887cb39ee5a144, State: Running, Role: LEADER
I20260923 17:22:07.999821 2724 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } }
I20260923 17:22:08.000397 2719 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:08.003751 2725 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } }
I20260923 17:22:08.004235 2725 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:08.004215 2726 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9e1d06ed1444e00ae887cb39ee5a144. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } }
I20260923 17:22:08.004581 2726 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:08.006213 2733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:08.012171 2733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:08.023280 2733 catalog_manager.cc:1347] Generated new cluster ID: fe89399da8a04b4db57b7b77d918fd64
I20260923 17:22:08.023471 2733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:08.056589 2733 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:22:08.058020 2733 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:08.075099 2733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144: Generated new TSK 0
I20260923 17:22:08.076287 2733 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.193.keytab.
Entry for principal oryx/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.193.keytab.
I20260923 17:22:08.153119 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:34367
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.193.keytab
--principal=oryx/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:08.534386 2745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:08.534754 2745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:08.534857 2745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:08.542383 2745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:08.542521 2745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:08.542639 2745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:22:08.552856 2745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.193.keytab
--principal=oryx/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:34367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:08.554194 2745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:08.555593 2745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:08.570055 2751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:08.570112 2750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:08.570072 2753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:08.571516 2745 server_base.cc:1034] running on GCE node
I20260923 17:22:08.572269 2745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:08.573367 2745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:08.574632 2745 hybrid_clock.cc:648] HybridClock initialized: now 1790184128574547 us; error 68 us; skew 500 ppm
I20260923 17:22:08.575078 2745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:08 dist-test-slave-j0m4 krb5kdc[2639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184128, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:08.578631 2745 init.cc:377] Logged in from keytab as oryx/127.28.72.193@KRBTEST.COM (short username oryx)
I20260923 17:22:08.580232 2745 webserver.cc:460] Webserver started at http://127.28.72.193:35797/ using document root <none> and password file <none>
I20260923 17:22:08.580986 2745 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:08.581146 2745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:08.581468 2745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:08.584594 2745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "3b1d20be2ecb4a1891207977144917e0"
format_stamp: "Formatted at 2026-09-23 17:22:08 on dist-test-slave-j0m4"
server_key: "2652335099f3fc7aa9e6fabd7df29a26"
server_key_iv: "731c932480e570606cd63123f55b6af4"
server_key_version: "encryptionkey@0"
I20260923 17:22:08.585410 2745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "3b1d20be2ecb4a1891207977144917e0"
format_stamp: "Formatted at 2026-09-23 17:22:08 on dist-test-slave-j0m4"
server_key: "2652335099f3fc7aa9e6fabd7df29a26"
server_key_iv: "731c932480e570606cd63123f55b6af4"
server_key_version: "encryptionkey@0"
I20260923 17:22:08.591024 2745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:22:08.594741 2760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:08.596449 2745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:08.596670 2745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "3b1d20be2ecb4a1891207977144917e0"
format_stamp: "Formatted at 2026-09-23 17:22:08 on dist-test-slave-j0m4"
server_key: "2652335099f3fc7aa9e6fabd7df29a26"
server_key_iv: "731c932480e570606cd63123f55b6af4"
server_key_version: "encryptionkey@0"
I20260923 17:22:08.596915 2745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:08.613795 2745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:08.618429 2745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:08.618741 2745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:08.619837 2745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:08.621243 2745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:08.621348 2745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:08.621541 2745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:08.621620 2745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:08.657339 2745 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:35277
I20260923 17:22:08.657384 2872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:35277 every 8 connection(s)
I20260923 17:22:08.658725 2745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:22:08.661659 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 2745
I20260923 17:22:08.661937 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:22:08.662953 28963 external_mini_cluster.cc:1428] Setting key 0c78197ab3d9d65083ccd09757d8b00c
Sep 23 17:22:08 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184128, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
I20260923 17:22:08.683306 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:08.661646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:57567 (local address 127.28.72.254:34367)
0923 17:22:08.663942 (+ 2296us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:08.663950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:08.663973 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:22:08.665690 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:08.665696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:08.665779 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:08.665911 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:08.667697 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:08.668653 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:08.670326 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:08.670827 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:08.670974 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:08.675264 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:08.675275 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:08.675284 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:08.675334 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:08.679376 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:08.680303 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:08.680312 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:08.680318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:08.680499 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:08.680990 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:08.680999 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:08.681005 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:08.681439 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:08.681615 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:08.682580 (+ 965us) server_negotiation.cc:299] Negotiation successful
0923 17:22:08.682845 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2169,"thread_start_us":2070,"threads_started":1}
I20260923 17:22:08.684410 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:08.662298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.193:57567)
0923 17:22:08.663243 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:08.663305 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:08.665402 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:08.666177 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:08.666186 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:08.666735 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:08.667505 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:08.667524 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:08.668834 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:08.668839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:08.670164 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:08.670178 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:08.671110 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:08.671173 (+ 63us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:08.671449 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:08.672269 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:08.672374 (+ 105us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:08.674941 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:08.679585 (+ 4644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:08.679594 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:08.679611 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:08.680123 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:08.680635 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:08.680639 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:08.680641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:08.680852 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:08.681667 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:08.681677 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:08.682079 (+ 402us) client_negotiation.cc:769] Sending connection context
0923 17:22:08.682404 (+ 325us) client_negotiation.cc:240] Negotiation successful
0923 17:22:08.682860 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":361,"threads_started":1}
WARNING: no policy specified for oryx/127.28.72.194@KRBTEST.COM; defaulting to no policy
I20260923 17:22:08.686149 2873 heartbeater.cc:347] Connected to a master server at 127.28.72.254:34367
I20260923 17:22:08.686465 2873 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:08.687361 2873 heartbeater.cc:511] Master 127.28.72.254:34367 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.72.194@KRBTEST.COM" created.
I20260923 17:22:08.689814 2684 ts_manager.cc:194] Registered new tserver with Master: 3b1d20be2ecb4a1891207977144917e0 (127.28.72.193:35277)
I20260923 17:22:08.691385 2684 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:57567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.194.keytab.
Entry for principal oryx/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.194.keytab.
I20260923 17:22:08.715251 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:34367
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.194.keytab
--principal=oryx/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:09.058718 2879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:09.059073 2879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:09.059273 2879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:09.066694 2879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:09.066880 2879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:09.067080 2879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:22:09.077442 2879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.194.keytab
--principal=oryx/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:34367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:09.078898 2879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:09.080389 2879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:09.094769 2884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:09.095077 2885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:09.095911 2887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:09.095968 2879 server_base.cc:1034] running on GCE node
I20260923 17:22:09.096690 2879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:09.097757 2879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:09.099037 2879 hybrid_clock.cc:648] HybridClock initialized: now 1790184129098980 us; error 49 us; skew 500 ppm
I20260923 17:22:09.099493 2879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:09 dist-test-slave-j0m4 krb5kdc[2639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184129, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:09.103487 2879 init.cc:377] Logged in from keytab as oryx/127.28.72.194@KRBTEST.COM (short username oryx)
I20260923 17:22:09.105722 2879 webserver.cc:460] Webserver started at http://127.28.72.194:36253/ using document root <none> and password file <none>
I20260923 17:22:09.106647 2879 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:09.106809 2879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:09.107188 2879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:09.110728 2879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "d19bba33d1f447cf94a81c91177d918f"
format_stamp: "Formatted at 2026-09-23 17:22:09 on dist-test-slave-j0m4"
server_key: "c92a9354af7e50f6242da3585bf90fd0"
server_key_iv: "9215a1dc4cd8e211bcc9eea97f94ef1f"
server_key_version: "encryptionkey@0"
I20260923 17:22:09.111663 2879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "d19bba33d1f447cf94a81c91177d918f"
format_stamp: "Formatted at 2026-09-23 17:22:09 on dist-test-slave-j0m4"
server_key: "c92a9354af7e50f6242da3585bf90fd0"
server_key_iv: "9215a1dc4cd8e211bcc9eea97f94ef1f"
server_key_version: "encryptionkey@0"
I20260923 17:22:09.117683 2879 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260923 17:22:09.122005 2894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:09.124115 2879 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260923 17:22:09.124365 2879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "d19bba33d1f447cf94a81c91177d918f"
format_stamp: "Formatted at 2026-09-23 17:22:09 on dist-test-slave-j0m4"
server_key: "c92a9354af7e50f6242da3585bf90fd0"
server_key_iv: "9215a1dc4cd8e211bcc9eea97f94ef1f"
server_key_version: "encryptionkey@0"
I20260923 17:22:09.124625 2879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:09.142638 2879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:09.147964 2879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:09.148312 2879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:09.149613 2879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:09.151218 2879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:09.151350 2879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:09.151511 2879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:09.151590 2879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:09.188863 2879 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:35325
I20260923 17:22:09.188925 3006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:35325 every 8 connection(s)
I20260923 17:22:09.190245 2879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:22:09.198559 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 2879
I20260923 17:22:09.198788 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:22:09.199381 28963 external_mini_cluster.cc:1428] Setting key e300b97e85547adc0e07897271d325fa
Sep 23 17:22:09 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184129, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
I20260923 17:22:09.214251 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:09.193888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:33589 (local address 127.28.72.254:34367)
0923 17:22:09.194400 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:09.194404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:09.195535 (+ 1131us) server_negotiation.cc:407] Connection header received
0923 17:22:09.197429 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:09.197435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:09.197527 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:09.197647 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:09.199626 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:09.200522 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:09.202658 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:09.203169 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:09.203305 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:09.207239 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:09.207244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:09.207248 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:09.207283 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:09.210368 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:09.211515 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:09.211520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:09.211522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:09.211669 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:09.212257 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:09.212261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:09.212263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:09.212640 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:09.212769 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:09.213727 (+ 958us) server_negotiation.cc:299] Negotiation successful
0923 17:22:09.213926 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":298,"threads_started":1}
I20260923 17:22:09.216007 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:09.194272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.194:33589)
0923 17:22:09.195234 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:09.195304 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:09.197106 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:09.197914 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:09.197940 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:09.198541 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:09.199394 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:09.199416 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:09.200697 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:09.200702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:09.202499 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:09.202515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:09.203431 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:09.203439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:09.203727 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:09.204550 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:09.204574 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:09.206911 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:09.210560 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:09.210575 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:09.210598 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:09.211346 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:09.211799 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:09.211806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:09.211813 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:09.212090 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:09.212790 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:09.212802 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:09.213312 (+ 510us) client_negotiation.cc:769] Sending connection context
0923 17:22:09.213621 (+ 309us) client_negotiation.cc:240] Negotiation successful
0923 17:22:09.214241 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":331,"threads_started":1}
I20260923 17:22:09.217856 3007 heartbeater.cc:347] Connected to a master server at 127.28.72.254:34367
I20260923 17:22:09.218231 3007 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:09.219264 3007 heartbeater.cc:511] Master 127.28.72.254:34367 requested a full tablet report, sending...
I20260923 17:22:09.221141 2684 ts_manager.cc:194] Registered new tserver with Master: d19bba33d1f447cf94a81c91177d918f (127.28.72.194:35325)
I20260923 17:22:09.222044 2684 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:33589
WARNING: no policy specified for oryx/127.28.72.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.72.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.195.keytab.
Entry for principal oryx/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.195.keytab.
I20260923 17:22:09.255514 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:34367
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.195.keytab
--principal=oryx/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:09.594439 3013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:09.594789 3013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:09.594954 3013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:09.602798 3013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:09.602948 3013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:09.603156 3013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:22:09.613618 3013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.195.keytab
--principal=oryx/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:34367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:09.615046 3013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:09.616456 3013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:09.630412 3021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:09.630412 3019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:09.630793 3018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:09.631321 3013 server_base.cc:1034] running on GCE node
I20260923 17:22:09.631968 3013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:09.633193 3013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:09.634440 3013 hybrid_clock.cc:648] HybridClock initialized: now 1790184129634402 us; error 28 us; skew 500 ppm
I20260923 17:22:09.634779 3013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:09 dist-test-slave-j0m4 krb5kdc[2639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184129, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:09.642530 3013 init.cc:377] Logged in from keytab as oryx/127.28.72.195@KRBTEST.COM (short username oryx)
I20260923 17:22:09.644263 3013 webserver.cc:460] Webserver started at http://127.28.72.195:43397/ using document root <none> and password file <none>
I20260923 17:22:09.645020 3013 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:09.645184 3013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:09.645503 3013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:09.648613 3013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "230eab516f7b4c07b7ef07e40a37cd35"
format_stamp: "Formatted at 2026-09-23 17:22:09 on dist-test-slave-j0m4"
server_key: "f405fdde3aecd7cffba57273c06e8d2e"
server_key_iv: "38b79c22f6e95a409d2351a8f258098b"
server_key_version: "encryptionkey@0"
I20260923 17:22:09.649420 3013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "230eab516f7b4c07b7ef07e40a37cd35"
format_stamp: "Formatted at 2026-09-23 17:22:09 on dist-test-slave-j0m4"
server_key: "f405fdde3aecd7cffba57273c06e8d2e"
server_key_iv: "38b79c22f6e95a409d2351a8f258098b"
server_key_version: "encryptionkey@0"
I20260923 17:22:09.654893 3013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:22:09.658869 3028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:09.660456 3013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:09.660641 3013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "230eab516f7b4c07b7ef07e40a37cd35"
format_stamp: "Formatted at 2026-09-23 17:22:09 on dist-test-slave-j0m4"
server_key: "f405fdde3aecd7cffba57273c06e8d2e"
server_key_iv: "38b79c22f6e95a409d2351a8f258098b"
server_key_version: "encryptionkey@0"
I20260923 17:22:09.660847 3013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:09.684724 3013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:09.689790 3013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:09.690062 3013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:09.691181 3013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:09.692612 3013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:09.692739 3013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:09.692909 3013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:09.692986 3013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:09.695011 2873 heartbeater.cc:503] Master 127.28.72.254:34367 was elected leader, sending a full tablet report...
I20260923 17:22:09.727774 3013 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:46417
I20260923 17:22:09.727833 3140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:46417 every 8 connection(s)
I20260923 17:22:09.729151 3013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:22:09.738523 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 3013
I20260923 17:22:09.738744 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:22:09.739228 28963 external_mini_cluster.cc:1428] Setting key de2fd7f410c6fde5d18f5859ea44a704
Sep 23 17:22:09 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184129, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
I20260923 17:22:09.750558 3143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:09.731931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:38227 (local address 127.28.72.254:34367)
0923 17:22:09.732415 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:09.732420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:09.733458 (+ 1038us) server_negotiation.cc:407] Connection header received
0923 17:22:09.735523 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:09.735529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:09.735627 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:09.735745 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:09.737275 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:09.738060 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:09.739881 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:09.740386 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:09.740520 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:09.744706 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:09.744712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:09.744715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:09.744750 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:09.747407 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:09.748146 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:09.748156 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:09.748159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:09.748268 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:09.748707 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:09.748712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:09.748717 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:09.748990 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:09.749175 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:09.749925 (+ 750us) server_negotiation.cc:299] Negotiation successful
0923 17:22:09.750158 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":258,"threads_started":1}
I20260923 17:22:09.751633 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:09.732520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.195:38227)
0923 17:22:09.733279 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:09.733318 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:09.735223 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:09.736013 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:09.736022 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:09.736542 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:09.737101 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:09.737122 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:09.738237 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:09.738241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:09.739724 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:09.739737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:09.740649 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:09.740663 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:09.740996 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:09.741928 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:09.741951 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:09.744462 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:09.747555 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:09.747565 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:09.747583 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:09.748014 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:09.748379 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:09.748384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:09.748387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:09.748587 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:09.749151 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:09.749159 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:09.749556 (+ 397us) client_negotiation.cc:769] Sending connection context
0923 17:22:09.749821 (+ 265us) client_negotiation.cc:240] Negotiation successful
0923 17:22:09.750366 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":295,"threads_started":1}
I20260923 17:22:09.753116 3141 heartbeater.cc:347] Connected to a master server at 127.28.72.254:34367
I20260923 17:22:09.753461 3141 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:09.754199 3141 heartbeater.cc:511] Master 127.28.72.254:34367 requested a full tablet report, sending...
I20260923 17:22:09.755738 2684 ts_manager.cc:194] Registered new tserver with Master: 230eab516f7b4c07b7ef07e40a37cd35 (127.28.72.195:46417)
I20260923 17:22:09.756570 2684 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:38227
I20260923 17:22:09.767709 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.72.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.72.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.72.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.253.keytab.
Entry for principal oryx/127.28.72.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.253.keytab.
I20260923 17:22:09.823868 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.253:33789
--webserver_interface=127.28.72.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.253.keytab
--principal=oryx/127.28.72.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:34367,127.28.72.253:33789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:10.177781 3147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:10.178102 3147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:10.178262 3147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:10.185761 3147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:10.185890 3147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:10.185988 3147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:10.186091 3147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:10.186179 3147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:10.198956 3147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:34367,127.28.72.253:33789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.253.keytab
--principal=oryx/127.28.72.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.253:33789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.72.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
W20260923 17:22:10.199448 3147 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.72.254:34367,127.28.72.253:33789'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 17:22:10.200623 3147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:10.202107 3147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:10.215220 3152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:10.215137 3153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:10.215200 3155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:10.215982 3147 server_base.cc:1034] running on GCE node
I20260923 17:22:10.216760 3147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:10.218346 3147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:10.219585 3147 hybrid_clock.cc:648] HybridClock initialized: now 1790184130219496 us; error 58 us; skew 500 ppm
I20260923 17:22:10.219992 3147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:10 dist-test-slave-j0m4 krb5kdc[2639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184130, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:10.223556 3147 init.cc:377] Logged in from keytab as oryx/127.28.72.253@KRBTEST.COM (short username oryx)
I20260923 17:22:10.225032 3007 heartbeater.cc:503] Master 127.28.72.254:34367 was elected leader, sending a full tablet report...
I20260923 17:22:10.225428 3147 webserver.cc:460] Webserver started at http://127.28.72.253:38731/ using document root <none> and password file <none>
I20260923 17:22:10.226361 3147 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:10.226459 3147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:10.226910 3147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:10.230206 3147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/data/instance:
uuid: "d1ad5c22fa4a47119df88053feb844f4"
format_stamp: "Formatted at 2026-09-23 17:22:10 on dist-test-slave-j0m4"
server_key: "2ffa35b04bc8db7a8a4e0d891eb50267"
server_key_iv: "7804c6c8f14817c02c1294e86cae590d"
server_key_version: "encryptionkey@0"
I20260923 17:22:10.231093 3147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/wal/instance:
uuid: "d1ad5c22fa4a47119df88053feb844f4"
format_stamp: "Formatted at 2026-09-23 17:22:10 on dist-test-slave-j0m4"
server_key: "2ffa35b04bc8db7a8a4e0d891eb50267"
server_key_iv: "7804c6c8f14817c02c1294e86cae590d"
server_key_version: "encryptionkey@0"
I20260923 17:22:10.237003 3147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:22:10.241024 3162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:10.242733 3147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:22:10.242940 3147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/wal
uuid: "d1ad5c22fa4a47119df88053feb844f4"
format_stamp: "Formatted at 2026-09-23 17:22:10 on dist-test-slave-j0m4"
server_key: "2ffa35b04bc8db7a8a4e0d891eb50267"
server_key_iv: "7804c6c8f14817c02c1294e86cae590d"
server_key_version: "encryptionkey@0"
I20260923 17:22:10.243227 3147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:10.290337 3147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:10.294854 3147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:10.295173 3147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:10.314595 3147 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.253:33789
I20260923 17:22:10.314601 3213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.253:33789 every 8 connection(s)
I20260923 17:22:10.316161 3147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/data/info.pb
I20260923 17:22:10.320286 3214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:10.326627 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 3147
I20260923 17:22:10.327106 3214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:22:10 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184130, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.253@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
Sep 23 17:22:10 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.72.253@KRBTEST.COM
I20260923 17:22:10.348407 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.330128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47754 (local address 127.28.72.254:34367)
0923 17:22:10.330780 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:10.330788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:10.331367 (+ 579us) server_negotiation.cc:407] Connection header received
0923 17:22:10.332817 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:10.332824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:10.332908 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:10.333041 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:10.334433 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.335263 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.336891 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.337280 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.337423 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:10.341151 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:10.341177 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:10.341186 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:10.341240 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:10.344225 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:10.345241 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:10.345246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:10.345248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:10.345458 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:10.345898 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:10.345904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:10.345908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:10.346358 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:10.346468 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:10.347544 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0923 17:22:10.347816 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":395,"threads_started":1}
I20260923 17:22:10.349520 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.329020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.28.72.253:33789)
0923 17:22:10.330178 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:10.330192 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:10.330257 (+ 65us) server_negotiation.cc:407] Connection header received
0923 17:22:10.332484 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:10.332507 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:10.332755 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:10.332972 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:10.334533 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.335801 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.337049 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.337583 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.337798 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:10.342051 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:10.342065 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:10.342088 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:10.342119 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:10.345441 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:10.346259 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:10.346266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:10.346268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:10.346403 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:10.346947 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:10.346951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:10.346954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:10.347421 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:10.347707 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:10.348092 (+ 385us) server_negotiation.cc:299] Negotiation successful
0923 17:22:10.348306 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":807,"thread_start_us":477,"threads_started":1}
I20260923 17:22:10.349531 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.330366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:47754)
0923 17:22:10.331219 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:10.331265 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:10.332516 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:10.333099 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:10.333110 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:10.333814 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:10.334190 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:10.334213 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.335412 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.335418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:10.336704 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:10.336716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.337565 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.337572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:10.337801 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:10.338604 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:10.338638 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:10.340932 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:10.344452 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:10.344463 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:10.344477 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:10.345095 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:10.345614 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:10.345617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:10.345619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:10.345783 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:10.346490 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:10.346548 (+ 58us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:10.346966 (+ 418us) client_negotiation.cc:769] Sending connection context
0923 17:22:10.347377 (+ 411us) client_negotiation.cc:240] Negotiation successful
0923 17:22:10.347995 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":3,"spinlock_wait_cycles":15616,"thread_start_us":446,"threads_started":1}
I20260923 17:22:10.351747 3214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:22:10 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184130, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.253@KRBTEST.COM for oryx/127.28.72.253@KRBTEST.COM
I20260923 17:22:10.365463 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.352473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:33789 (local address 127.0.0.1:43062)
0923 17:22:10.352618 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:10.352633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:10.352714 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:10.353443 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:10.353446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:10.353760 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:10.353925 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:10.353933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.355017 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.355020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:10.356014 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:10.356025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.356783 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.356787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:10.356918 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:10.357575 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:10.357592 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:10.359773 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:10.362737 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:10.362741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:10.362744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:10.363105 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:10.363712 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:10.363715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:10.363717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:10.363845 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:10.364650 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:10.364655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:10.364790 (+ 135us) client_negotiation.cc:769] Sending connection context
0923 17:22:10.364904 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 17:22:10.365148 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 17:22:10.365711 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.352983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.28.72.253:33789)
0923 17:22:10.353148 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:10.353153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:10.353166 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:22:10.353203 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:10.353206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:10.353254 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:10.353416 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:10.354053 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.354868 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.356161 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.356647 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.356760 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:10.359969 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:10.359974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:10.359977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:10.360006 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:10.362591 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:10.363292 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:10.363300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:10.363307 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:10.363579 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:10.363993 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:10.363998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:10.364000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:10.364470 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:10.364601 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:10.365024 (+ 423us) server_negotiation.cc:299] Negotiation successful
0923 17:22:10.365391 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":53}
I20260923 17:22:10.370507 3214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: Bootstrap starting.
I20260923 17:22:10.373711 3214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:10.374786 3214 log.cc:824] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:10.377574 3214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: No bootstrap required, opened a new log
I20260923 17:22:10.384748 3214 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } }
I20260923 17:22:10.385154 3214 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:10.385265 3214 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1ad5c22fa4a47119df88053feb844f4, State: Initialized, Role: FOLLOWER
I20260923 17:22:10.385921 3214 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } }
I20260923 17:22:10.386847 3221 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } } }
I20260923 17:22:10.387292 3221 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:22:10.387964 3214 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:22:10.396872 3232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:22:10.397030 3232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:22:10.399686 3147 master_runner.cc:428] Detected that this master d1ad5c22fa4a47119df88053feb844f4 is joining an existing cluster
I20260923 17:22:10.399797 3147 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.72.253:33789
I20260923 17:22:10.422834 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.404155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.28.72.254:34367)
0923 17:22:10.404309 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:10.404314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:10.405973 (+ 1659us) server_negotiation.cc:407] Connection header received
0923 17:22:10.406221 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:10.406227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:10.406318 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:10.406388 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:10.407981 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.409126 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.412273 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.412966 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.413078 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:10.416030 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:10.416038 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:10.416043 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:10.416085 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:10.419375 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:10.420274 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:10.420281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:10.420284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:10.420442 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:10.420925 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:10.420933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:10.420939 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:10.421311 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:10.421469 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:10.422115 (+ 646us) server_negotiation.cc:299] Negotiation successful
0923 17:22:10.422375 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 17:22:10.424151 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.404304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:47756)
0923 17:22:10.405873 (+ 1569us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:10.405891 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:10.406077 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:10.406618 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:10.406627 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:10.407365 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:10.407800 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:10.407813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.410362 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.410369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:10.412091 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:10.412113 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.413382 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.413389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:10.413592 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:10.414571 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:10.414604 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:10.415796 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:10.419533 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:10.419541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:10.419548 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:10.420136 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:10.420586 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:10.420593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:10.420600 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:10.420795 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:10.421551 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:10.421557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:10.421805 (+ 248us) client_negotiation.cc:769] Sending connection context
0923 17:22:10.423153 (+ 1348us) client_negotiation.cc:240] Negotiation successful
0923 17:22:10.423553 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1421,"thread_start_us":12996,"threads_started":1}
I20260923 17:22:10.434091 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.404806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.28.72.253:33789)
0923 17:22:10.404992 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:10.404998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:10.417767 (+ 12769us) server_negotiation.cc:407] Connection header received
0923 17:22:10.418014 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:10.418019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:10.418100 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:10.418196 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:10.419552 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.420678 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.422673 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.423446 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.423564 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:10.426020 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:10.426027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:10.426031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:10.426063 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:10.429827 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:10.431836 (+ 2009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:10.431841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:10.431845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:10.432012 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:10.432436 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:10.432440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:10.432444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:10.432841 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:10.433000 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:10.433469 (+ 469us) server_negotiation.cc:299] Negotiation successful
0923 17:22:10.433705 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 17:22:10.435309 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.405195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:33789 (local address 127.0.0.1:43070)
0923 17:22:10.417658 (+ 12463us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:10.417674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:10.417839 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:10.418357 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:10.418362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:10.418999 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:10.419387 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:10.419399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.420892 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.420896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:10.422520 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:10.422536 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.423872 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.423882 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:10.424052 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:10.424712 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:10.424734 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:10.425819 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:10.430435 (+ 4616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:10.430440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:10.430444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:10.431711 (+ 1267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:10.432155 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:10.432160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:10.432165 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:10.432321 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:10.433068 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:10.433074 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:10.433229 (+ 155us) client_negotiation.cc:769] Sending connection context
0923 17:22:10.434500 (+ 1271us) client_negotiation.cc:240] Negotiation successful
0923 17:22:10.434803 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12314,"thread_start_us":368,"threads_started":1}
WARNING: no policy specified for oryx/127.28.72.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.72.252@KRBTEST.COM" created.
Sep 23 17:22:10 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184127, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.254@KRBTEST.COM for oryx/127.28.72.253@KRBTEST.COM
I20260923 17:22:10.456080 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.433601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.28.72.253:33789)
0923 17:22:10.434456 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:10.434463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:10.439080 (+ 4617us) server_negotiation.cc:407] Connection header received
0923 17:22:10.439370 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:10.439378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:10.439485 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:10.439594 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:10.440992 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.442159 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.444517 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.445350 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.445480 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:10.450038 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:10.450043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:10.450046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:10.450077 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:10.453043 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:10.453799 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:10.453807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:10.453815 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:10.453947 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:10.454454 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:10.454457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:10.454459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:10.454745 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:10.454898 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:10.455433 (+ 535us) server_negotiation.cc:299] Negotiation successful
0923 17:22:10.455675 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":5872,"threads_started":1}
I20260923 17:22:10.456123 3241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.433234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:33789 (local address 127.0.0.1:43078)
0923 17:22:10.438922 (+ 5688us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:10.438960 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:10.439167 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:10.439837 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:10.439843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:10.440586 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:10.440829 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:10.440839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.442311 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.442327 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:10.444337 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:10.444356 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.445765 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.445773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:10.446036 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:10.446855 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:10.446883 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:10.449812 (+ 2929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:10.453193 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:10.453209 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:10.453217 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:10.453671 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:10.454068 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:10.454074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:10.454078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:10.454317 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:10.454875 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:10.454885 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:10.455075 (+ 190us) client_negotiation.cc:769] Sending connection context
0923 17:22:10.455337 (+ 262us) client_negotiation.cc:240] Negotiation successful
0923 17:22:10.455703 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5428,"thread_start_us":5271,"threads_started":1}
I20260923 17:22:10.458302 2684 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.72.253:33789: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4"
member_type: NON_VOTER
last_known_addr {
host: "127.28.72.253"
port: 33789
}
attrs {
promote: true
}
}
dest_uuid: "a9e1d06ed1444e00ae887cb39ee5a144"
cas_config_opid_index: -1
I20260923 17:22:10.459316 2684 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: NON_VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: true } }
I20260923 17:22:10.466423 3245 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d1ad5c22fa4a47119df88053feb844f4 (127.28.72.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: NON_VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: true } } }
I20260923 17:22:10.467600 3189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:10.471194 3189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 FOLLOWER]: Refusing update from remote peer a9e1d06ed1444e00ae887cb39ee5a144: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 17:22:10.471400 3245 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.72.253:33789
I20260923 17:22:10.472262 3147 master.cc:537] Master@127.28.72.253:33789 shutting down...
I20260923 17:22:10.472369 3245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: NON_VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:10.473757 3246 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: NON_VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.72.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.252.keytab.
Entry for principal oryx/127.28.72.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.252.keytab.
I20260923 17:22:10.475273 3246 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: This master's current role is: LEADER
W20260923 17:22:10.475842 2670 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260923 17:22:10.477147 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.252:43907
--webserver_interface=127.28.72.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.252.keytab
--principal=oryx/127.28.72.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:34367,127.28.72.253:33789,127.28.72.252:43907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:10.477670 2670 consensus_peers.cc:487] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 -> Peer d1ad5c22fa4a47119df88053feb844f4 (127.28.72.253:33789): Couldn't send request to peer d1ad5c22fa4a47119df88053feb844f4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 17:22:10.487654 3147 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:10.488147 3147 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:10.488356 3147 tablet_replica.cc:330] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: stopping tablet replica
I20260923 17:22:10.495026 3147 master.cc:552] Master@127.28.72.253:33789 shutdown complete.
I20260923 17:22:10.495100 3147 master_runner.cc:305] Clearing existing system tablet
I20260923 17:22:10.495492 3147 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:10.495558 3147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:10.499800 3147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260923 17:22:10.501749 3251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:10.502056 3147 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:10.502344 3147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:22:10.502425 3147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/wal
uuid: "d1ad5c22fa4a47119df88053feb844f4"
format_stamp: "Formatted at 2026-09-23 17:22:10 on dist-test-slave-j0m4"
server_key: "2ffa35b04bc8db7a8a4e0d891eb50267"
server_key_iv: "7804c6c8f14817c02c1294e86cae590d"
server_key_version: "encryptionkey@0"
I20260923 17:22:10.506160 3147 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:10.511502 3147 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 17:22:10.511704 3147 log.cc:1196] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 17:22:10.512145 3147 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: Deleting consensus metadata
I20260923 17:22:10.512409 3147 master_runner.cc:315] Copying system tablet from 127.28.72.254:34367
I20260923 17:22:10.514972 3147 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: tablet copy: Beginning tablet copy session from remote peer at address 127.28.72.254:34367
I20260923 17:22:10.527849 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.515877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:47764)
0923 17:22:10.516485 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:10.516507 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:10.516669 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:10.517053 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:10.517057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:10.517476 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:10.517736 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:10.517750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.518775 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.518784 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:10.519825 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:10.519837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.520577 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.520582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:10.520800 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:10.521447 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:10.521471 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:10.522227 (+ 756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:10.525265 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:10.525270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:10.525275 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:10.525646 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:10.526026 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:10.526031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:10.526035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:10.526166 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:10.526702 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:10.526707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:10.526889 (+ 182us) client_negotiation.cc:769] Sending connection context
0923 17:22:10.527058 (+ 169us) client_negotiation.cc:240] Negotiation successful
0923 17:22:10.527383 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":368,"threads_started":1}
I20260923 17:22:10.527884 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.515884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.28.72.254:34367)
0923 17:22:10.516105 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:10.516114 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:10.516599 (+ 485us) server_negotiation.cc:407] Connection header received
0923 17:22:10.516827 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:10.516833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:10.516907 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:10.517016 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:10.517894 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.518624 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.519966 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:10.520410 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:10.520563 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:10.522424 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:10.522431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:10.522443 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:10.522477 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:10.525118 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:10.525772 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:10.525777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:10.525781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:10.525910 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:10.526271 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:10.526278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:10.526283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:10.526577 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:10.526738 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:10.527172 (+ 434us) server_negotiation.cc:299] Negotiation successful
0923 17:22:10.527388 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:22:10.528885 2704 tablet_copy_service.cc:140] P a9e1d06ed1444e00ae887cb39ee5a144: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d1ad5c22fa4a47119df88053feb844f4 ({username='oryx', principal='oryx/127.28.72.253@KRBTEST.COM'} at 127.0.0.1:47764)
I20260923 17:22:10.529156 2704 tablet_copy_service.cc:161] P a9e1d06ed1444e00ae887cb39ee5a144: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d1ad5c22fa4a47119df88053feb844f4 at {username='oryx', principal='oryx/127.28.72.253@KRBTEST.COM'} at 127.0.0.1:47764: session id = d1ad5c22fa4a47119df88053feb844f4-00000000000000000000000000000000
I20260923 17:22:10.532490 2704 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 17:22:10.534439 3147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 17:22:10.539539 3147 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: tablet copy: Starting download of 0 data blocks...
I20260923 17:22:10.539886 3147 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: tablet copy: Starting download of 1 WAL segments...
I20260923 17:22:10.542493 3147 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 17:22:10.546108 2704 tablet_copy_service.cc:342] P a9e1d06ed1444e00ae887cb39ee5a144: Request end of tablet copy session d1ad5c22fa4a47119df88053feb844f4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.72.253@KRBTEST.COM'} at 127.0.0.1:47764
I20260923 17:22:10.546334 2704 tablet_copy_service.cc:433] P a9e1d06ed1444e00ae887cb39ee5a144: ending tablet copy session d1ad5c22fa4a47119df88053feb844f4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d1ad5c22fa4a47119df88053feb844f4
I20260923 17:22:10.557056 3147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:10.559937 3258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:10.566254 3259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:10.571617 3147 server_base.cc:1034] running on GCE node
W20260923 17:22:10.572057 3261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:10.572556 3147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:10.573233 3147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:10.574507 3147 hybrid_clock.cc:648] HybridClock initialized: now 1790184130574398 us; error 42 us; skew 500 ppm
I20260923 17:22:10.574746 3147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:10 dist-test-slave-j0m4 krb5kdc[2639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184130, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:10.576692 3147 init.cc:377] Logged in from keytab as oryx/127.28.72.253@KRBTEST.COM (short username oryx)
I20260923 17:22:10.578011 3147 webserver.cc:460] Webserver started at http://127.28.72.253:35351/ using document root <none> and password file <none>
I20260923 17:22:10.578347 3147 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:10.578444 3147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:10.581032 3147 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:22:10.582706 3268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:10.583254 3147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260923 17:22:10.583401 3147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/wal
uuid: "d1ad5c22fa4a47119df88053feb844f4"
format_stamp: "Formatted at 2026-09-23 17:22:10 on dist-test-slave-j0m4"
server_key: "2ffa35b04bc8db7a8a4e0d891eb50267"
server_key_iv: "7804c6c8f14817c02c1294e86cae590d"
server_key_version: "encryptionkey@0"
I20260923 17:22:10.583573 3147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:10.608795 3147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:10.612393 3147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:10.642184 3147 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.253:33789
I20260923 17:22:10.642206 3319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.253:33789 every 8 connection(s)
I20260923 17:22:10.643265 3147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-1/data/info.pb
I20260923 17:22:10.650609 3320 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:22:10.653407 3320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: Bootstrap starting.
I20260923 17:22:10.669363 3320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:22:10.669893 3320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: Bootstrap complete.
I20260923 17:22:10.670919 3320 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: NON_VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: true } }
I20260923 17:22:10.671183 3320 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1ad5c22fa4a47119df88053feb844f4, State: Initialized, Role: LEARNER
I20260923 17:22:10.671420 3320 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: NON_VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: true } }
I20260923 17:22:10.672271 3323 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: NON_VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: true } } }
I20260923 17:22:10.672623 3320 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:10.672672 3323 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:22:10.683364 3334 catalog_manager.cc:1259] Loaded cluster ID: fe89399da8a04b4db57b7b77d918fd64
I20260923 17:22:10.683547 3334 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: loading cluster ID for follower catalog manager: success
I20260923 17:22:10.687282 3334 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: acquiring CA information for follower catalog manager: success
I20260923 17:22:10.689940 3334 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:22:10.759194 3141 heartbeater.cc:503] Master 127.28.72.254:34367 was elected leader, sending a full tablet report...
W20260923 17:22:10.896939 3249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:10.897255 3249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:10.897346 3249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:10.905921 3249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:10.906049 3249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:10.906095 3249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:10.906152 3249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:10.906196 3249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:10.916699 3249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:34367,127.28.72.253:33789,127.28.72.252:43907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.252.keytab
--principal=oryx/127.28.72.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.252:43907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.72.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:10.918045 3249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:10.919368 3249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:10.932438 3342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:10.932464 3339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:10.932395 3340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:10.932912 3249 server_base.cc:1034] running on GCE node
I20260923 17:22:10.934018 3249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:10.935631 3249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:10.936914 3249 hybrid_clock.cc:648] HybridClock initialized: now 1790184130936871 us; error 41 us; skew 500 ppm
I20260923 17:22:10.937289 3249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:10 dist-test-slave-j0m4 krb5kdc[2639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184130, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:10.940703 3249 init.cc:377] Logged in from keytab as oryx/127.28.72.252@KRBTEST.COM (short username oryx)
I20260923 17:22:10.942226 3249 webserver.cc:460] Webserver started at http://127.28.72.252:41995/ using document root <none> and password file <none>
I20260923 17:22:10.942973 3249 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:10.943084 3249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:10.943408 3249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:10.946555 3249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/data/instance:
uuid: "c3b8453220e145eeaba2927bf0cc938c"
format_stamp: "Formatted at 2026-09-23 17:22:10 on dist-test-slave-j0m4"
server_key: "42bb501667c644979ed595ef365a646a"
server_key_iv: "10a00e0659b4e6b72be83af7b1e199ec"
server_key_version: "encryptionkey@0"
I20260923 17:22:10.947386 3249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/wal/instance:
uuid: "c3b8453220e145eeaba2927bf0cc938c"
format_stamp: "Formatted at 2026-09-23 17:22:10 on dist-test-slave-j0m4"
server_key: "42bb501667c644979ed595ef365a646a"
server_key_iv: "10a00e0659b4e6b72be83af7b1e199ec"
server_key_version: "encryptionkey@0"
I20260923 17:22:10.953485 3249 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260923 17:22:10.958084 3349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:10.959863 3249 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260923 17:22:10.960043 3249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/wal
uuid: "c3b8453220e145eeaba2927bf0cc938c"
format_stamp: "Formatted at 2026-09-23 17:22:10 on dist-test-slave-j0m4"
server_key: "42bb501667c644979ed595ef365a646a"
server_key_iv: "10a00e0659b4e6b72be83af7b1e199ec"
server_key_version: "encryptionkey@0"
I20260923 17:22:10.960265 3249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:10.981379 3249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:10.987078 3249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:10.987461 3249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:11.008810 3249 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.252:43907
I20260923 17:22:11.008880 3400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.252:43907 every 8 connection(s)
I20260923 17:22:11.010542 3249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/data/info.pb
I20260923 17:22:11.015028 3401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:11.017573 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 3249
I20260923 17:22:11.023963 3401 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:22:11 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.72.252@KRBTEST.COM
Sep 23 17:22:11 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184130, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.252@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
I20260923 17:22:11.043569 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.027096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47778 (local address 127.28.72.254:34367)
0923 17:22:11.027324 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.027332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.027946 (+ 614us) server_negotiation.cc:407] Connection header received
0923 17:22:11.028121 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.028126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.028200 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.028309 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:11.029935 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.030963 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.033012 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.033612 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.033754 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.037484 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:11.037489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:11.037493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:11.037527 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:11.040212 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.040951 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.040959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.040967 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.041093 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.041545 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.041550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.041559 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.041827 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:11.041988 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.042786 (+ 798us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.043020 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260923 17:22:11.044497 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.026989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:47778)
0923 17:22:11.027791 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.027842 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:11.027992 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:11.028363 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:11.028368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:11.029450 (+ 1082us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:11.029777 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.029787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.031107 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.031136 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.032799 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.032816 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.033839 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.033844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.033980 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.034608 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:11.034631 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:11.037273 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:11.040387 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.040400 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.040413 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.040810 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.041203 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.041209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.041213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.041400 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.041962 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:11.041970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:11.042432 (+ 462us) client_negotiation.cc:769] Sending connection context
0923 17:22:11.042694 (+ 262us) client_negotiation.cc:240] Negotiation successful
0923 17:22:11.043242 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"spinlock_wait_cycles":28416,"thread_start_us":384,"threads_started":1}
I20260923 17:22:11.044513 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.019943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.28.72.252:43907)
0923 17:22:11.020787 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.020806 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.020861 (+ 55us) server_negotiation.cc:407] Connection header received
0923 17:22:11.023271 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.023302 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.023863 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.024328 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:11.027022 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.028796 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.030781 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.031961 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.032162 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.037002 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:11.037014 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:11.037030 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:11.037081 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:11.040167 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.041287 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.041302 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.041304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.041444 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.041816 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.041823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.041826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.042343 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:11.042648 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.042979 (+ 331us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.043388 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":585,"thread_start_us":321,"threads_started":1}
I20260923 17:22:11.046619 3401 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:22:11 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184130, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.252@KRBTEST.COM for oryx/127.28.72.253@KRBTEST.COM
I20260923 17:22:11.062479 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.047586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:33789 (local address 127.0.0.1:43090)
0923 17:22:11.047783 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.047804 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:11.047954 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:11.048499 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:11.048503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:11.048901 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:11.049042 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.049050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.050334 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.050341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.051340 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.051353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.052082 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.052088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.052272 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.052916 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:11.052934 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:11.055210 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:11.059816 (+ 4606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.059821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.059825 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.060425 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.060823 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.060828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.060832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.060988 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.061692 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:11.061695 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:11.061819 (+ 124us) client_negotiation.cc:769] Sending connection context
0923 17:22:11.061941 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 17:22:11.062171 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 17:22:11.062683 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.047531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.28.72.253:33789)
0923 17:22:11.048213 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.048217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.048231 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:22:11.048289 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.048292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.048360 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.048475 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:11.049191 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.050195 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.051484 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.051900 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.052068 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.055415 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:11.055421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:11.055424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:11.055581 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:11.059666 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.060553 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.060557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.060559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.060709 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.061125 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.061129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.061132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.061565 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:11.061721 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.062090 (+ 369us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.062307 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":422,"threads_started":1}
I20260923 17:22:11.063997 3401 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:22:11.065809 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.057655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:33789 (local address 127.0.0.1:43104)
0923 17:22:11.059268 (+ 1613us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.059284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:11.059453 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:11.059874 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:11.059878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:11.059900 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:11.060044 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.060051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.061473 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.061480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.063464 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.063489 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.064984 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.064990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.065155 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.065229 (+ 74us) client_negotiation.cc:769] Sending connection context
0923 17:22:11.065347 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 17:22:11.065495 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1506,"thread_start_us":1359,"threads_started":1}
I20260923 17:22:11.066031 3409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.057989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43104 (local address 127.28.72.253:33789)
0923 17:22:11.059152 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.059160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.059371 (+ 211us) server_negotiation.cc:407] Connection header received
0923 17:22:11.059586 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.059591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.059667 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.059798 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:11.060184 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.061317 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.063618 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.064798 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.064945 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.065203 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.065382 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.065510 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":955,"thread_start_us":809,"threads_started":1}
I20260923 17:22:11.067004 3295 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260923 17:22:11.067649 3323 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9e1d06ed1444e00ae887cb39ee5a144. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: NON_VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: true } } }
I20260923 17:22:11.067910 3323 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: This master's current role is: LEARNER
Sep 23 17:22:11 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184130, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.252@KRBTEST.COM for oryx/127.28.72.252@KRBTEST.COM
I20260923 17:22:11.078120 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.064953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43907 (local address 127.0.0.1:50408)
0923 17:22:11.065238 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.065257 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:11.065387 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:11.065726 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:11.065730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:11.066241 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:11.066480 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.066494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.067549 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.067553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.068561 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.068573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.069281 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.069285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.069474 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.070079 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:11.070097 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:11.072326 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:11.075673 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.075681 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.075689 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.076087 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.076523 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.076530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.076533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.076750 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.077287 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:11.077291 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:11.077486 (+ 195us) client_negotiation.cc:769] Sending connection context
0923 17:22:11.077612 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 17:22:11.077821 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260923 17:22:11.078667 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.064861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.28.72.252:43907)
0923 17:22:11.065046 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.065052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.065326 (+ 274us) server_negotiation.cc:407] Connection header received
0923 17:22:11.065543 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.065549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.065618 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.065711 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:11.066625 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.067384 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.068722 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.069141 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.069242 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.072522 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:11.072533 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:11.072537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:11.072567 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:11.075535 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.076213 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.076217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.076219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.076343 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.076888 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.076892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.076894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.077173 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:11.077286 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.077712 (+ 426us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.077955 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":74}
I20260923 17:22:11.084002 3401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: Bootstrap starting.
I20260923 17:22:11.087419 3401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:11.088539 3401 log.cc:824] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:11.091738 3401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: No bootstrap required, opened a new log
I20260923 17:22:11.100785 3401 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } }
I20260923 17:22:11.101245 3401 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:11.101390 3401 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b8453220e145eeaba2927bf0cc938c, State: Initialized, Role: FOLLOWER
I20260923 17:22:11.102121 3401 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } }
I20260923 17:22:11.103080 3411 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } } }
I20260923 17:22:11.103650 3411 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:22:11.104346 3401 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:22:11.114969 3422 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:22:11.115160 3422 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:22:11.120404 3249 master_runner.cc:428] Detected that this master c3b8453220e145eeaba2927bf0cc938c is joining an existing cluster
I20260923 17:22:11.120558 3249 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.72.252:43907
I20260923 17:22:11.147533 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.125462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47792 (local address 127.28.72.254:34367)
0923 17:22:11.125688 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.125703 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.129701 (+ 3998us) server_negotiation.cc:407] Connection header received
0923 17:22:11.129934 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.129940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.130054 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.130170 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:11.131966 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.133162 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.136140 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.136794 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.136905 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.140081 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:11.140088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:11.140093 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:11.140135 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:11.144065 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.145039 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.145045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.145049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.145209 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.145695 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.145700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.145704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.146078 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:11.146223 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.146875 (+ 652us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.147146 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:22:11.148118 3409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.125780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.28.72.253:33789)
0923 17:22:11.126084 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.126090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.130637 (+ 4547us) server_negotiation.cc:407] Connection header received
0923 17:22:11.130808 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.130813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.130892 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.131005 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:11.132507 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.133648 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.136648 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.137312 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.137412 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.140732 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:11.140739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:11.140744 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:11.140793 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:11.144871 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.145706 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.145711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.145715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.145861 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.146292 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.146299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.146305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.146685 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:11.146809 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.147388 (+ 579us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.147800 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260923 17:22:11.148696 3428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.125901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:47792)
0923 17:22:11.129510 (+ 3609us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.129542 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:11.129768 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:11.130420 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:11.130430 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:11.131328 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:11.131765 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.131779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.133410 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.133416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.135954 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.135969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.137203 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.137209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.137385 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.138314 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:11.138337 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:11.139830 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:11.144248 (+ 4418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.144253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.144257 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.144885 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.145355 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.145359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.145363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.145558 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.146303 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:11.146308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:11.146531 (+ 223us) client_negotiation.cc:769] Sending connection context
0923 17:22:11.147828 (+ 1297us) client_negotiation.cc:240] Negotiation successful
0923 17:22:11.148196 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3372,"mutex_wait_us":1,"thread_start_us":3157,"threads_started":1}
I20260923 17:22:11.149212 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.126848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:33789 (local address 127.0.0.1:43118)
0923 17:22:11.130510 (+ 3662us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.130538 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:11.130683 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:11.131023 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:11.131027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:11.131940 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:11.132301 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.132313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.133824 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.133831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.136478 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.136498 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.137710 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.137723 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.137898 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.139078 (+ 1180us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:11.139096 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:11.140504 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:11.145040 (+ 4536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.145046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.145051 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.145566 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.146005 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.146009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.146012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.146170 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.146891 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:11.146895 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:11.147090 (+ 195us) client_negotiation.cc:769] Sending connection context
0923 17:22:11.148339 (+ 1249us) client_negotiation.cc:240] Negotiation successful
0923 17:22:11.148697 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3533,"mutex_wait_us":30,"spinlock_wait_cycles":176384,"thread_start_us":3336,"threads_started":1}
I20260923 17:22:11.151053 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.125945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.28.72.252:43907)
0923 17:22:11.126178 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.126185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.131820 (+ 5635us) server_negotiation.cc:407] Connection header received
0923 17:22:11.132031 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.132038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.132102 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.132178 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:11.133491 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.135320 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.137187 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.137955 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.138060 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.140690 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:11.140695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:11.140699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:11.140735 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:11.145346 (+ 4611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.146125 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.146129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.146132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.146241 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.146595 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.146600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.146603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.147034 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:11.147187 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.147918 (+ 731us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.149236 (+ 1318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1098,"server-negotiator.queue_time_us":91}
I20260923 17:22:11.151438 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 2745
I20260923 17:22:11.153154 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.130679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43907 (local address 127.0.0.1:50412)
0923 17:22:11.131728 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.131751 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:11.131903 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:11.132353 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:11.132359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:11.133002 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:11.133336 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.133347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.135538 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.135543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.137052 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.137065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.138130 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.138138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.138329 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.138939 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:11.138972 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:11.140504 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:11.145500 (+ 4996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.145506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.145510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.145998 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.146357 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.146362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.146366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.146487 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.147270 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:11.147277 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:11.147438 (+ 161us) client_negotiation.cc:769] Sending connection context
0923 17:22:11.148024 (+ 586us) client_negotiation.cc:240] Negotiation successful
0923 17:22:11.152267 (+ 4243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":547,"spinlock_wait_cycles":1920,"thread_start_us":3643,"threads_started":1}
Sep 23 17:22:11 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184127, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.254@KRBTEST.COM for oryx/127.28.72.252@KRBTEST.COM
I20260923 17:22:11.169795 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.155057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43907 (local address 127.0.0.1:50418)
0923 17:22:11.155323 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.155342 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:11.155474 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:11.155895 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:11.155902 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:11.156296 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:11.156438 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.156446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.157449 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.157458 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.158408 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.158421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.159217 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.159223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.159426 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.160054 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:11.160071 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:11.162200 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:11.166760 (+ 4560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.166766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.166771 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.167401 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.167847 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.167852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.167857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.168034 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.168706 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:11.168712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:11.168910 (+ 198us) client_negotiation.cc:769] Sending connection context
0923 17:22:11.169070 (+ 160us) client_negotiation.cc:240] Negotiation successful
0923 17:22:11.169393 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260923 17:22:11.169843 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.155104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.28.72.252:43907)
0923 17:22:11.155280 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.155285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.155421 (+ 136us) server_negotiation.cc:407] Connection header received
0923 17:22:11.155634 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.155639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.155754 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.155874 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:11.156573 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.157314 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.158552 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.159036 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.159208 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.162404 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:11.162409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:11.162412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:11.162442 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:11.166582 (+ 4140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.167547 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.167553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.167557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.167713 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.168180 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.168185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.168189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.168567 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:11.168718 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.169228 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.169514 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 17:22:11.171842 2684 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.72.252:43907: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c"
member_type: NON_VOTER
last_known_addr {
host: "127.28.72.252"
port: 43907
}
attrs {
promote: true
}
}
dest_uuid: "a9e1d06ed1444e00ae887cb39ee5a144"
cas_config_opid_index: 5
I20260923 17:22:11.172962 2684 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: NON_VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: true } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } }
I20260923 17:22:11.175011 3407 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c3b8453220e145eeaba2927bf0cc938c (127.28.72.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: NON_VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: true } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } } }
I20260923 17:22:11.175907 3376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:11.180049 3376 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 1 FOLLOWER]: Refusing update from remote peer a9e1d06ed1444e00ae887cb39ee5a144: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 17:22:11.180512 3295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 LEARNER]: Refusing update from remote peer a9e1d06ed1444e00ae887cb39ee5a144: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 17:22:11.179080 3407 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.72.252:43907
Sep 23 17:22:11 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.72.193@KRBTEST.COM
I20260923 17:22:11.194023 3407 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: NON_VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 17:22:11.194243 3432 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: NON_VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: true } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } } }
I20260923 17:22:11.194792 3432 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:11.202047 3432 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:11.206535 3249 master.cc:537] Master@127.28.72.252:43907 shutting down...
I20260923 17:22:11.211999 3295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c3b8453220e145eeaba2927bf0cc938c (127.28.72.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: NON_VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: true } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } } }
I20260923 17:22:11.215433 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.152849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.28.72.193:35277)
0923 17:22:11.163695 (+ 10846us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.163707 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.163739 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:22:11.163809 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.163819 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.164163 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.164499 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:11.166041 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.167710 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.169877 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.170887 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.171041 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.189761 (+ 18720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:11.189790 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:11.189821 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:11.189892 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:11.209452 (+ 19560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.211881 (+ 2429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.211891 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.211894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.212107 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.212607 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.212614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.212620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.213098 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:11.214465 (+ 1367us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.214656 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.214947 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10635,"thread_start_us":10439,"threads_started":1}
W20260923 17:22:11.216945 2671 consensus_peers.cc:487] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 -> Peer c3b8453220e145eeaba2927bf0cc938c (127.28.72.252:43907): Couldn't send request to peer c3b8453220e145eeaba2927bf0cc938c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 17:22:11.217168 2868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:11.217689 3323 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: NON_VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: true } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } } }
I20260923 17:22:11.218035 3323 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:22:11.218502 3432 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144: attempting to promote NON_VOTER d1ad5c22fa4a47119df88053feb844f4 to VOTER
I20260923 17:22:11.219632 3432 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } }
I20260923 17:22:11.223618 3295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 LEARNER]: Refusing update from remote peer a9e1d06ed1444e00ae887cb39ee5a144: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 17:22:11.224335 3407 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 17:22:11.225912 3249 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:11.226490 3249 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:11.226804 3249 tablet_replica.cc:330] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: stopping tablet replica
W20260923 17:22:11.227049 2671 consensus_peers.cc:487] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 -> Peer c3b8453220e145eeaba2927bf0cc938c (127.28.72.252:43907): Couldn't send request to peer c3b8453220e145eeaba2927bf0cc938c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 17:22:11.227931 3434 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d1ad5c22fa4a47119df88053feb844f4 (127.28.72.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } } }
I20260923 17:22:11.228853 3295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d1ad5c22fa4a47119df88053feb844f4 (127.28.72.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } } }
I20260923 17:22:11.232369 3407 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } } }
I20260923 17:22:11.232642 3407 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:11.233326 3434 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } } }
I20260923 17:22:11.233644 3434 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:11.233980 3323 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } } }
I20260923 17:22:11.234259 3323 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:22:11.237514 3249 master.cc:552] Master@127.28.72.252:43907 shutdown complete.
I20260923 17:22:11.237720 3249 master_runner.cc:305] Clearing existing system tablet
I20260923 17:22:11.238212 3249 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:11.238514 3249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:11.242142 3249 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:22:11.244339 3440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:11.244827 3249 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:11.245339 3249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:22:11.245515 3249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/wal
uuid: "c3b8453220e145eeaba2927bf0cc938c"
format_stamp: "Formatted at 2026-09-23 17:22:10 on dist-test-slave-j0m4"
server_key: "42bb501667c644979ed595ef365a646a"
server_key_iv: "10a00e0659b4e6b72be83af7b1e199ec"
server_key_version: "encryptionkey@0"
I20260923 17:22:11.252041 3249 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:11.260427 3249 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 17:22:11.260743 3249 log.cc:1196] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 17:22:11.261426 3249 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: Deleting consensus metadata
I20260923 17:22:11.261760 3249 master_runner.cc:315] Copying system tablet from 127.28.72.254:34367
I20260923 17:22:11.264631 3249 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.72.254:34367
I20260923 17:22:11.284090 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.265598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.28.72.254:34367)
0923 17:22:11.265839 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.265846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.267784 (+ 1938us) server_negotiation.cc:407] Connection header received
0923 17:22:11.268093 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.268100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.268207 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.268328 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:11.269533 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.270661 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.272814 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.273584 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.273734 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.276558 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:11.276566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:11.276571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:11.276639 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:11.280744 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.281737 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.281743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.281747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.281911 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.282389 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.282395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.282398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.282776 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:11.282947 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.283456 (+ 509us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.283709 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260923 17:22:11.284678 3446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.265546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:47796)
0923 17:22:11.267642 (+ 2096us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.267669 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:11.267912 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:11.268387 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:11.268393 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:11.269022 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:11.269341 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.269355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.270851 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.270857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.272625 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.272642 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.273768 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.273775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.274051 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.274889 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:11.274919 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:11.276282 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:11.280953 (+ 4671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.280960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.280965 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.281572 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.282047 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.282053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.282058 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.282253 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.282921 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:11.282927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:11.283164 (+ 237us) client_negotiation.cc:769] Sending connection context
0923 17:22:11.283320 (+ 156us) client_negotiation.cc:240] Negotiation successful
0923 17:22:11.283772 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1932,"thread_start_us":1764,"threads_started":1}
I20260923 17:22:11.285586 2704 tablet_copy_service.cc:140] P a9e1d06ed1444e00ae887cb39ee5a144: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c3b8453220e145eeaba2927bf0cc938c ({username='oryx', principal='oryx/127.28.72.252@KRBTEST.COM'} at 127.0.0.1:47796)
I20260923 17:22:11.285769 2704 tablet_copy_service.cc:161] P a9e1d06ed1444e00ae887cb39ee5a144: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c3b8453220e145eeaba2927bf0cc938c at {username='oryx', principal='oryx/127.28.72.252@KRBTEST.COM'} at 127.0.0.1:47796: session id = c3b8453220e145eeaba2927bf0cc938c-00000000000000000000000000000000
I20260923 17:22:11.289577 2704 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 17:22:11.291771 3249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 17:22:11.300170 3249 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: tablet copy: Starting download of 0 data blocks...
I20260923 17:22:11.300611 3249 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: tablet copy: Starting download of 1 WAL segments...
I20260923 17:22:11.303452 3249 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 17:22:11.308795 2704 tablet_copy_service.cc:342] P a9e1d06ed1444e00ae887cb39ee5a144: Request end of tablet copy session c3b8453220e145eeaba2927bf0cc938c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.72.252@KRBTEST.COM'} at 127.0.0.1:47796
I20260923 17:22:11.308974 2704 tablet_copy_service.cc:433] P a9e1d06ed1444e00ae887cb39ee5a144: ending tablet copy session c3b8453220e145eeaba2927bf0cc938c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c3b8453220e145eeaba2927bf0cc938c
I20260923 17:22:11.321835 3249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:11.325022 3447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:11.328668 3448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:11.336503 3249 server_base.cc:1034] running on GCE node
W20260923 17:22:11.336899 3450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:11.337423 3249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:11.339394 3249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:11.340806 3249 hybrid_clock.cc:648] HybridClock initialized: now 1790184131340743 us; error 51 us; skew 500 ppm
I20260923 17:22:11.341168 3249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:11 dist-test-slave-j0m4 krb5kdc[2639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184131, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:11.343801 3249 init.cc:377] Logged in from keytab as oryx/127.28.72.252@KRBTEST.COM (short username oryx)
I20260923 17:22:11.345336 3249 webserver.cc:460] Webserver started at http://127.28.72.252:36883/ using document root <none> and password file <none>
I20260923 17:22:11.345692 3249 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:11.345800 3249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:11.349493 3249 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:22:11.351725 3457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:11.352258 3249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:22:11.352392 3249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/wal
uuid: "c3b8453220e145eeaba2927bf0cc938c"
format_stamp: "Formatted at 2026-09-23 17:22:10 on dist-test-slave-j0m4"
server_key: "42bb501667c644979ed595ef365a646a"
server_key_iv: "10a00e0659b4e6b72be83af7b1e199ec"
server_key_version: "encryptionkey@0"
I20260923 17:22:11.352577 3249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:11.365854 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 2745
I20260923 17:22:11.376116 3249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:11.380615 3249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:11.385058 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:35277
--local_ip_for_outbound_sockets=127.28.72.193
--tserver_master_addrs=127.28.72.254:34367
--webserver_port=35797
--webserver_interface=127.28.72.193
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.193.keytab
--principal=oryx/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.72.254:34367,127.28.72.253:33789
--tserver_master_addrs=127.28.72.254:34367,127.28.72.253:33789,127.28.72.252:43907 with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:22:11.421053 3249 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.252:43907
I20260923 17:22:11.421084 3509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.252:43907 every 8 connection(s)
I20260923 17:22:11.422163 3249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/master-2/data/info.pb
I20260923 17:22:11.427249 3510 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:22:11.431588 3510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: Bootstrap starting.
I20260923 17:22:11.459956 3510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:22:11.460505 3510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: Bootstrap complete.
I20260923 17:22:11.461566 3510 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } }
I20260923 17:22:11.461886 3510 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c3b8453220e145eeaba2927bf0cc938c, State: Initialized, Role: LEARNER
I20260923 17:22:11.462150 3510 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } }
I20260923 17:22:11.463028 3513 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } } }
I20260923 17:22:11.463331 3510 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:11.463564 3513 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [sys.catalog]: This master's current role is: LEARNER
I20260923 17:22:11.477227 3524 catalog_manager.cc:1259] Loaded cluster ID: fe89399da8a04b4db57b7b77d918fd64
I20260923 17:22:11.477339 3524 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: loading cluster ID for follower catalog manager: success
I20260923 17:22:11.480908 3524 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: acquiring CA information for follower catalog manager: success
I20260923 17:22:11.483670 3524 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:22:11.715570 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.708976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43907 (local address 127.0.0.1:50434)
0923 17:22:11.709488 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.709513 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:11.709667 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:11.710147 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:11.710152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:11.710181 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:11.710416 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.710426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.711813 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.711821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.713002 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.713015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.714180 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.714186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.714889 (+ 703us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.714930 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 17:22:11.715044 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 17:22:11.715219 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":269,"threads_started":1}
I20260923 17:22:11.715579 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.709232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.28.72.252:43907)
0923 17:22:11.709826 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.709830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.709845 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:22:11.709903 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.709907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.709974 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.710101 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:11.710605 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.711669 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.713168 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.714002 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.714142 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.714816 (+ 674us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.715068 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.715225 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":328,"threads_started":1}
I20260923 17:22:11.716791 3485 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 17:22:11.717521 3513 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9e1d06ed1444e00ae887cb39ee5a144. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } } }
I20260923 17:22:11.717926 3513 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [sys.catalog]: This master's current role is: LEARNER
W20260923 17:22:11.776647 3465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:11.777012 3465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:11.777176 3465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:11.785053 3465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:11.785216 3465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:11.785403 3465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:22:11.796267 3465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.193.keytab
--principal=oryx/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:35277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=35797
--tserver_master_addrs=127.28.72.254:34367,127.28.72.253:33789,127.28.72.252:43907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:11.797807 3465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:11.799387 3465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:11.814440 3532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:11.814535 3534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:11.815305 3531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:11.815699 3465 server_base.cc:1034] running on GCE node
I20260923 17:22:11.816408 3465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:11.817561 3465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:11.818815 3465 hybrid_clock.cc:648] HybridClock initialized: now 1790184131818747 us; error 48 us; skew 500 ppm
I20260923 17:22:11.819375 3465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:11 dist-test-slave-j0m4 krb5kdc[2639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184131, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:11.822726 3465 init.cc:377] Logged in from keytab as oryx/127.28.72.193@KRBTEST.COM (short username oryx)
I20260923 17:22:11.824455 3465 webserver.cc:460] Webserver started at http://127.28.72.193:35797/ using document root <none> and password file <none>
I20260923 17:22:11.825201 3465 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:11.825362 3465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:11.831616 3465 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:22:11.834649 3541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:11.836284 3465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:11.836556 3465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "3b1d20be2ecb4a1891207977144917e0"
format_stamp: "Formatted at 2026-09-23 17:22:08 on dist-test-slave-j0m4"
server_key: "2652335099f3fc7aa9e6fabd7df29a26"
server_key_iv: "731c932480e570606cd63123f55b6af4"
server_key_version: "encryptionkey@0"
I20260923 17:22:11.837612 3465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:11.852161 3465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:11.856760 3465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:11.857100 3465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:11.858143 3465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:11.859745 3465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:11.859869 3465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:11.860016 3465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:11.860112 3465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:11.903023 3465 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:35277
I20260923 17:22:11.903120 3653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:35277 every 8 connection(s)
I20260923 17:22:11.904870 3465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:22:11.914608 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 3465
I20260923 17:22:11.915036 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 2879
Sep 23 17:22:11 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184131, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.252@KRBTEST.COM
Sep 23 17:22:11 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184131, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
Sep 23 17:22:11 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184131, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.253@KRBTEST.COM
I20260923 17:22:11.944156 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.910059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:39621 (local address 127.28.72.252:43907)
0923 17:22:11.910278 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.910287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.923718 (+ 13431us) server_negotiation.cc:407] Connection header received
0923 17:22:11.924251 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.924256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.924350 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.924432 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:11.925789 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.926797 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.928744 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.929463 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.929566 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.935364 (+ 5798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:11.935663 (+ 299us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:11.935670 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:11.935723 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:11.939691 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.940928 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.940936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.940942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.941185 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.941815 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.941821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.941824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.942270 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:11.942424 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.943528 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.943769 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 17:22:11.946745 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.909024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:39475 (local address 127.28.72.254:34367)
0923 17:22:11.909492 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.909498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.921617 (+ 12119us) server_negotiation.cc:407] Connection header received
0923 17:22:11.921726 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.921735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.921859 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.922023 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:11.926885 (+ 4862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.928142 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.930055 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.930658 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.930792 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.939141 (+ 8349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:11.939151 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:11.939157 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:11.939207 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:11.943233 (+ 4026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.944047 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.944052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.944055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.944198 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.944634 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.944639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.944641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.945067 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:11.945199 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.946159 (+ 960us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.946394 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":221,"threads_started":1}
I20260923 17:22:11.947499 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.910088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.193:39475)
0923 17:22:11.915894 (+ 5806us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.915921 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:11.916086 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:11.922379 (+ 6293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:11.922387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:11.923649 (+ 1262us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:11.924009 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.924024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.928347 (+ 4323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.928352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.929860 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.929891 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.931072 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.931081 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.931302 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.932808 (+ 1506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:11.932842 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:11.938802 (+ 5960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:11.943397 (+ 4595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.943405 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.943411 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.943893 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.944347 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.944351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.944354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.944511 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.945268 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:11.945272 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:11.945443 (+ 171us) client_negotiation.cc:769] Sending connection context
0923 17:22:11.945600 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 17:22:11.945862 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5621,"thread_start_us":392,"threads_started":1}
I20260923 17:22:11.947526 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.910143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43907 (local address 127.28.72.193:39621)
0923 17:22:11.923589 (+ 13446us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.923614 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:11.924098 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:11.924635 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:11.924640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:11.925255 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:11.925609 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.925624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.926984 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.926989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.928560 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.928573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.929955 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.929965 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.930275 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.931207 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:11.931241 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:11.934980 (+ 3739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:11.939997 (+ 5017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.940009 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.940036 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.940761 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.941352 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.941357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.941362 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.941675 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.942523 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:11.942534 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:11.943038 (+ 504us) client_negotiation.cc:769] Sending connection context
0923 17:22:11.944873 (+ 1835us) client_negotiation.cc:240] Negotiation successful
0923 17:22:11.945539 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13287,"spinlock_wait_cycles":22272,"thread_start_us":5345,"threads_started":1}
I20260923 17:22:11.949106 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.908625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:58569 (local address 127.28.72.253:33789)
0923 17:22:11.909183 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.909191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.911274 (+ 2083us) server_negotiation.cc:407] Connection header received
0923 17:22:11.916662 (+ 5388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.916672 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.916815 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.916957 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:11.920619 (+ 3662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.922143 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.931675 (+ 9532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.932429 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.932538 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.940326 (+ 7788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:11.940337 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:11.940345 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:11.940420 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:11.944681 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.945873 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.945883 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.945887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.946076 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.947269 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.947275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.947278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.947692 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:11.947834 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.948439 (+ 605us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.948703 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":542,"threads_started":1}
I20260923 17:22:11.949461 3654 heartbeater.cc:347] Connected to a master server at 127.28.72.252:43907
I20260923 17:22:11.949807 3654 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:11.949817 3655 heartbeater.cc:347] Connected to a master server at 127.28.72.254:34367
I20260923 17:22:11.949978 3655 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:11.950874 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.909109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:33789 (local address 127.28.72.193:58569)
0923 17:22:11.911030 (+ 1921us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.911094 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:11.914880 (+ 3786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:11.917664 (+ 2784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:11.917684 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:11.919076 (+ 1392us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:11.920277 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.920316 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.924243 (+ 3927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.924248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.926680 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:11.926695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.934906 (+ 8211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.934914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:11.935137 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.936773 (+ 1636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:11.936801 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:11.940000 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:11.944927 (+ 4927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.944934 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.944941 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.945680 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.946264 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:11.946271 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:11.946278 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:11.946501 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:11.947930 (+ 1429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:11.947934 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:11.948144 (+ 210us) client_negotiation.cc:769] Sending connection context
0923 17:22:11.949341 (+ 1197us) client_negotiation.cc:240] Negotiation successful
0923 17:22:11.949675 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1500,"thread_start_us":14455,"threads_started":1}
I20260923 17:22:11.950928 3654 heartbeater.cc:511] Master 127.28.72.252:43907 requested a full tablet report, sending...
I20260923 17:22:11.951421 3655 heartbeater.cc:511] Master 127.28.72.254:34367 requested a full tablet report, sending...
I20260923 17:22:11.951875 3656 heartbeater.cc:347] Connected to a master server at 127.28.72.253:33789
I20260923 17:22:11.952008 3656 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:11.952471 3656 heartbeater.cc:511] Master 127.28.72.253:33789 requested a full tablet report, sending...
I20260923 17:22:11.952946 2684 ts_manager.cc:194] Re-registered known tserver with Master: 3b1d20be2ecb4a1891207977144917e0 (127.28.72.193:35277)
I20260923 17:22:11.953606 3475 ts_manager.cc:194] Registered new tserver with Master: 3b1d20be2ecb4a1891207977144917e0 (127.28.72.193:35277)
I20260923 17:22:11.953972 2684 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:39475
I20260923 17:22:11.956717 3285 ts_manager.cc:194] Registered new tserver with Master: 3b1d20be2ecb4a1891207977144917e0 (127.28.72.193:35277)
Sep 23 17:22:11 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.72.194@KRBTEST.COM
I20260923 17:22:11.971364 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.916157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.28.72.194:35325)
0923 17:22:11.947047 (+ 30890us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:11.947056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:11.947085 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:22:11.947172 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:11.947181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:11.947487 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:11.949508 (+ 2021us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:11.950622 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.955482 (+ 4860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.957326 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:11.958170 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:11.958302 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:11.963525 (+ 5223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:11.963543 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:11.963564 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:11.963618 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:11.967992 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.968911 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.968921 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.968925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.969093 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:11.969604 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:11.969608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:11.969614 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:11.970053 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:11.970328 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:11.970643 (+ 315us) server_negotiation.cc:299] Negotiation successful
0923 17:22:11.970920 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30706,"thread_start_us":30496,"threads_started":1}
I20260923 17:22:11.972862 3002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:12.062196 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 2879
I20260923 17:22:12.079618 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:35325
--local_ip_for_outbound_sockets=127.28.72.194
--tserver_master_addrs=127.28.72.254:34367
--webserver_port=36253
--webserver_interface=127.28.72.194
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.194.keytab
--principal=oryx/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.72.254:34367,127.28.72.253:33789
--tserver_master_addrs=127.28.72.254:34367,127.28.72.253:33789,127.28.72.252:43907 with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:22:12.122803 3407 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: NON_VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: true } } }
I20260923 17:22:12.123260 3407 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:12.124929 3407 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144: attempting to promote NON_VOTER c3b8453220e145eeaba2927bf0cc938c to VOTER
I20260923 17:22:12.125527 3407 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } }
I20260923 17:22:12.127389 3295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 FOLLOWER]: Refusing update from remote peer a9e1d06ed1444e00ae887cb39ee5a144: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 17:22:12.127408 3485 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 1 LEARNER]: Refusing update from remote peer a9e1d06ed1444e00ae887cb39ee5a144: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 17:22:12.128055 3407 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 17:22:12.128476 3668 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 17:22:12.131639 3295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c3b8453220e145eeaba2927bf0cc938c (127.28.72.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } } }
I20260923 17:22:12.133991 3485 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c3b8453220e145eeaba2927bf0cc938c (127.28.72.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } } }
I20260923 17:22:12.130929 3407 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c3b8453220e145eeaba2927bf0cc938c (127.28.72.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } } }
I20260923 17:22:12.138096 3407 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } } }
I20260923 17:22:12.138365 3407 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:12.138230 3668 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } } }
I20260923 17:22:12.138473 3668 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:12.138474 3513 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } } }
I20260923 17:22:12.138738 3513 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:22:12.138629 3668 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } } }
I20260923 17:22:12.138917 3668 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:12.142391 3671 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } } }
I20260923 17:22:12.142890 3671 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 17:22:12.425304 3666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:12.425632 3666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:12.425812 3666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:12.433420 3666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:12.433629 3666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:12.433812 3666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:22:12.444238 3666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.194.keytab
--principal=oryx/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:35325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=36253
--tserver_master_addrs=127.28.72.254:34367,127.28.72.253:33789,127.28.72.252:43907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:12.445664 3666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:12.447016 3666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:12.461122 3676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:12.461166 3677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:12.461347 3679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:12.461938 3666 server_base.cc:1034] running on GCE node
I20260923 17:22:12.462636 3666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:12.463837 3666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:12.465103 3666 hybrid_clock.cc:648] HybridClock initialized: now 1790184132465032 us; error 62 us; skew 500 ppm
I20260923 17:22:12.465516 3666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:12 dist-test-slave-j0m4 krb5kdc[2639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184132, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:12.469390 3666 init.cc:377] Logged in from keytab as oryx/127.28.72.194@KRBTEST.COM (short username oryx)
I20260923 17:22:12.471212 3666 webserver.cc:460] Webserver started at http://127.28.72.194:36253/ using document root <none> and password file <none>
I20260923 17:22:12.472078 3666 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:12.472263 3666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:12.478225 3666 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:22:12.481400 3686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:12.483282 3666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:12.483563 3666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "d19bba33d1f447cf94a81c91177d918f"
format_stamp: "Formatted at 2026-09-23 17:22:09 on dist-test-slave-j0m4"
server_key: "c92a9354af7e50f6242da3585bf90fd0"
server_key_iv: "9215a1dc4cd8e211bcc9eea97f94ef1f"
server_key_version: "encryptionkey@0"
I20260923 17:22:12.484828 3666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:12.528527 3666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:12.533620 3666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:12.533895 3666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:12.534915 3666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:12.536520 3666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:12.536617 3666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:12.536697 3666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:12.536744 3666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:12.573369 3666 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:35325
I20260923 17:22:12.573395 3798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:35325 every 8 connection(s)
I20260923 17:22:12.575029 3666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:22:12.584039 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 3666
I20260923 17:22:12.584357 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 3013
Sep 23 17:22:12 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184132, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.253@KRBTEST.COM
I20260923 17:22:12.608735 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:12.579681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:55025 (local address 127.28.72.253:33789)
0923 17:22:12.580216 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:12.580223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:12.580245 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:22:12.584260 (+ 4015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:12.584268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:12.584374 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:12.584472 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:12.587413 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:12.588913 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:12.592392 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:12.593164 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:12.593285 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:12.599537 (+ 6252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:12.599548 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:12.599555 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:12.599609 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:12.604171 (+ 4562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:12.605390 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:12.605399 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:12.605405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:12.605575 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:12.606178 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:12.606183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:12.606187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:12.606606 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:12.606765 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:12.607979 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0923 17:22:12.608218 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":436,"threads_started":1}
I20260923 17:22:12.612170 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:12.578969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:33789 (local address 127.28.72.194:55025)
0923 17:22:12.580055 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:12.580117 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:12.583467 (+ 3350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:12.585139 (+ 1672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:12.585153 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:12.586028 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:12.587141 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:12.587176 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:12.589185 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:12.589193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:12.592165 (+ 2972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:12.592186 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:12.593733 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:12.593743 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:12.594130 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:12.595153 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:12.595197 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:12.599173 (+ 3976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:12.604390 (+ 5217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:12.604405 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:12.604431 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:12.605223 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:12.605739 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:12.605747 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:12.605753 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:12.606035 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:12.606890 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:12.606901 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:12.607474 (+ 573us) client_negotiation.cc:769] Sending connection context
0923 17:22:12.609281 (+ 1807us) client_negotiation.cc:240] Negotiation successful
0923 17:22:12.610080 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":1041,"threads_started":1}
I20260923 17:22:12.614867 3801 heartbeater.cc:347] Connected to a master server at 127.28.72.253:33789
I20260923 17:22:12.615268 3801 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:12.616223 3801 heartbeater.cc:511] Master 127.28.72.253:33789 requested a full tablet report, sending...
I20260923 17:22:12.618186 3285 ts_manager.cc:194] Registered new tserver with Master: d19bba33d1f447cf94a81c91177d918f (127.28.72.194:35325)
Sep 23 17:22:12 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184132, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
Sep 23 17:22:12 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184132, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.252@KRBTEST.COM
Sep 23 17:22:12 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.72.195@KRBTEST.COM
I20260923 17:22:12.632251 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:12.578564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:47397 (local address 127.28.72.254:34367)
0923 17:22:12.605092 (+ 26528us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:12.605101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:12.613477 (+ 8376us) server_negotiation.cc:407] Connection header received
0923 17:22:12.613747 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:12.613753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:12.613869 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:12.613949 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:12.615370 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:12.616415 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:12.618329 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:12.618950 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:12.619063 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:12.625677 (+ 6614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:12.625686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:12.625692 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:12.625739 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:12.629113 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:12.629871 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:12.629876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:12.629880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:12.630055 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:12.630622 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:12.630627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:12.630630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:12.630952 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:12.631142 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:12.631597 (+ 455us) server_negotiation.cc:299] Negotiation successful
0923 17:22:12.631855 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26352,"thread_start_us":26562,"threads_started":1}
I20260923 17:22:12.632335 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:12.578969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.194:47397)
0923 17:22:12.613375 (+ 34406us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:12.613393 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:12.613594 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:12.614161 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:12.614166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:12.614868 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:12.615181 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:12.615193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:12.616622 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:12.616627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:12.618158 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:12.618174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:12.619384 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:12.619390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:12.619563 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:12.621956 (+ 2393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:12.622007 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:12.624813 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:12.629256 (+ 4443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:12.629262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:12.629266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:12.629736 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:12.630363 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:12.630367 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:12.630369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:12.630506 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:12.631090 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:12.631098 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:12.631311 (+ 213us) client_negotiation.cc:769] Sending connection context
0923 17:22:12.631456 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 17:22:12.631791 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34219,"mutex_wait_us":903,"thread_start_us":33074,"threads_started":1}
I20260923 17:22:12.633147 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:12.580542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43907 (local address 127.28.72.194:42293)
0923 17:22:12.615557 (+ 35015us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:12.615583 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:12.615856 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:12.616644 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:12.616651 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:12.617355 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:12.617692 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:12.617708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:12.619911 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:12.619919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:12.621700 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:12.621711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:12.622586 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:12.622592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:12.622779 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:12.623508 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:12.623534 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:12.626113 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:12.630320 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:12.630325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:12.630329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:12.630812 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:12.631243 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:12.631246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:12.631249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:12.631465 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:12.632113 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:12.632117 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:12.632292 (+ 175us) client_negotiation.cc:769] Sending connection context
0923 17:22:12.632498 (+ 206us) client_negotiation.cc:240] Negotiation successful
0923 17:22:12.632767 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34663,"mutex_wait_us":3,"spinlock_wait_cycles":24960,"thread_start_us":39504,"threads_started":1}
I20260923 17:22:12.633663 3800 heartbeater.cc:347] Connected to a master server at 127.28.72.254:34367
I20260923 17:22:12.633826 3800 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:12.634099 3803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:12.578267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:42293 (local address 127.28.72.252:43907)
0923 17:22:12.607651 (+ 29384us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:12.607660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:12.615692 (+ 8032us) server_negotiation.cc:407] Connection header received
0923 17:22:12.616040 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:12.616049 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:12.616153 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:12.616249 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:12.617939 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:12.619659 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:12.621869 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:12.622428 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:12.622563 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:12.626320 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:12.626326 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:12.626329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:12.626363 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:12.630147 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:12.630941 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:12.630946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:12.630949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:12.631090 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:12.631590 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:12.631595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:12.631598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:12.631917 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:12.632044 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:12.633524 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0923 17:22:12.633759 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29219,"thread_start_us":29388,"threads_started":1}
I20260923 17:22:12.634348 3800 heartbeater.cc:511] Master 127.28.72.254:34367 requested a full tablet report, sending...
I20260923 17:22:12.635277 3799 heartbeater.cc:347] Connected to a master server at 127.28.72.252:43907
I20260923 17:22:12.635437 3799 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:12.635468 2684 ts_manager.cc:194] Re-registered known tserver with Master: d19bba33d1f447cf94a81c91177d918f (127.28.72.194:35325)
I20260923 17:22:12.635834 3799 heartbeater.cc:511] Master 127.28.72.252:43907 requested a full tablet report, sending...
I20260923 17:22:12.636271 2684 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:47397
I20260923 17:22:12.637094 3475 ts_manager.cc:194] Registered new tserver with Master: d19bba33d1f447cf94a81c91177d918f (127.28.72.194:35325)
I20260923 17:22:12.639876 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:12.585911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.28.72.195:46417)
0923 17:22:12.609210 (+ 23299us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:12.609222 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:12.609255 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:22:12.609354 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:12.609364 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:12.609662 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:12.609904 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:12.613420 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:12.620519 (+ 7099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:12.622220 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:12.623401 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:12.623542 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:12.630837 (+ 7295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:12.630858 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:12.630883 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:12.630932 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:12.634925 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:12.637494 (+ 2569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:12.637502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:12.637504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:12.637658 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:12.638078 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:12.638084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:12.638087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:12.638440 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:12.638745 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:12.639104 (+ 359us) server_negotiation.cc:299] Negotiation successful
0923 17:22:12.639424 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23114,"thread_start_us":22938,"threads_started":1}
I20260923 17:22:12.641109 3136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:12.736132 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 3013
I20260923 17:22:12.752985 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:46417
--local_ip_for_outbound_sockets=127.28.72.195
--tserver_master_addrs=127.28.72.254:34367
--webserver_port=43397
--webserver_interface=127.28.72.195
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.195.keytab
--principal=oryx/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.72.254:34367,127.28.72.253:33789
--tserver_master_addrs=127.28.72.254:34367,127.28.72.253:33789,127.28.72.252:43907 with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:22:12.957031 3655 heartbeater.cc:503] Master 127.28.72.254:34367 was elected leader, sending a full tablet report...
W20260923 17:22:13.107069 3812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:13.107463 3812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:13.107626 3812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:13.115346 3812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:13.115515 3812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:13.115691 3812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:22:13.126180 3812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.195.keytab
--principal=oryx/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:46417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=43397
--tserver_master_addrs=127.28.72.254:34367,127.28.72.253:33789,127.28.72.252:43907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:13.127714 3812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:13.129066 3812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:13.143919 3818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:13.143970 3820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:13.143966 3817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:13.145313 3812 server_base.cc:1034] running on GCE node
I20260923 17:22:13.146003 3812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:13.147039 3812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:13.148312 3812 hybrid_clock.cc:648] HybridClock initialized: now 1790184133148249 us; error 57 us; skew 500 ppm
I20260923 17:22:13.148663 3812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:13 dist-test-slave-j0m4 krb5kdc[2639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184133, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:13.152020 3812 init.cc:377] Logged in from keytab as oryx/127.28.72.195@KRBTEST.COM (short username oryx)
I20260923 17:22:13.153631 3812 webserver.cc:460] Webserver started at http://127.28.72.195:43397/ using document root <none> and password file <none>
I20260923 17:22:13.154521 3812 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:13.154664 3812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:13.160583 3812 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:22:13.163728 3827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:13.165377 3812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 17:22:13.165660 3812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "230eab516f7b4c07b7ef07e40a37cd35"
format_stamp: "Formatted at 2026-09-23 17:22:09 on dist-test-slave-j0m4"
server_key: "f405fdde3aecd7cffba57273c06e8d2e"
server_key_iv: "38b79c22f6e95a409d2351a8f258098b"
server_key_version: "encryptionkey@0"
I20260923 17:22:13.166749 3812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:13.187052 3812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:13.191663 3812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:13.191968 3812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:13.192975 3812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:13.194506 3812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:13.194597 3812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:13.194717 3812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:13.194811 3812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:13.233207 3812 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:46417
I20260923 17:22:13.233266 3939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:46417 every 8 connection(s)
I20260923 17:22:13.234786 3812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:22:13.236475 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 3812
Sep 23 17:22:13 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184133, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.252@KRBTEST.COM
Sep 23 17:22:13 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184133, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.253@KRBTEST.COM
Sep 23 17:22:13 dist-test-slave-j0m4 krb5kdc[2639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184133, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
I20260923 17:22:13.287391 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.239340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:39957 (local address 127.28.72.253:33789)
0923 17:22:13.246471 (+ 7131us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.246481 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.252243 (+ 5762us) server_negotiation.cc:407] Connection header received
0923 17:22:13.260426 (+ 8183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.260436 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.260558 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.260651 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:13.265726 (+ 5075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.267061 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.271375 (+ 4314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.272192 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.272317 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.278791 (+ 6474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:13.278799 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:13.278804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:13.278848 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:13.282786 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.284180 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.284184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.284186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.284319 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.284964 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.284969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.284972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.285383 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:13.285562 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.286673 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.286940 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6960,"thread_start_us":6841,"threads_started":1}
I20260923 17:22:13.289588 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.238613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:33789 (local address 127.28.72.195:39957)
0923 17:22:13.252049 (+ 13436us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:13.252118 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:13.256358 (+ 4240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:13.260903 (+ 4545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:13.260910 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:13.261434 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:13.261766 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.261779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.267328 (+ 5549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.267336 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.271048 (+ 3712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.271065 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.273116 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.273125 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.273401 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.274220 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:13.274254 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:13.278509 (+ 4255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:13.282961 (+ 4452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:13.282972 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:13.282994 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:13.283700 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:13.284448 (+ 748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:13.284456 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:13.284462 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:13.284752 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:13.285537 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:13.285552 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:13.286175 (+ 623us) client_negotiation.cc:769] Sending connection context
0923 17:22:13.286548 (+ 373us) client_negotiation.cc:240] Negotiation successful
0923 17:22:13.287287 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13002,"mutex_wait_us":1534,"thread_start_us":23457,"threads_started":1}
I20260923 17:22:13.291767 3942 heartbeater.cc:347] Connected to a master server at 127.28.72.253:33789
I20260923 17:22:13.292129 3942 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:13.293223 3942 heartbeater.cc:511] Master 127.28.72.253:33789 requested a full tablet report, sending...
I20260923 17:22:13.294355 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.237847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:33433 (local address 127.28.72.252:43907)
0923 17:22:13.250657 (+ 12810us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.250667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.252874 (+ 2207us) server_negotiation.cc:407] Connection header received
0923 17:22:13.256953 (+ 4079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.256963 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.257072 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.257164 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:13.261993 (+ 4829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.263177 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.269597 (+ 6420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.270677 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.270896 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.278915 (+ 8019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:13.278926 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:13.278935 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:13.278999 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:13.284500 (+ 5501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.291882 (+ 7382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.291889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.291893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.292074 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.292598 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.292603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.292606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.292955 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:13.293086 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.293641 (+ 555us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.293932 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12654,"thread_start_us":12795,"threads_started":1}
I20260923 17:22:13.294385 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.239774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43907 (local address 127.28.72.195:33433)
0923 17:22:13.252733 (+ 12959us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:13.252754 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:13.256770 (+ 4016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:13.257838 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:13.257853 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:13.259074 (+ 1221us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:13.260138 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.260167 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.263404 (+ 3237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.263412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.269361 (+ 5949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.269393 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.271373 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.271385 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.271858 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.273005 (+ 1147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:13.273052 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:13.278559 (+ 5507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:13.284731 (+ 6172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:13.284745 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:13.284753 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:13.287365 (+ 2612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:13.292245 (+ 4880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:13.292250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:13.292253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:13.292441 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:13.293157 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:13.293160 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:13.293295 (+ 135us) client_negotiation.cc:769] Sending connection context
0923 17:22:13.293519 (+ 224us) client_negotiation.cc:240] Negotiation successful
0923 17:22:13.293878 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12804,"mutex_wait_us":87,"thread_start_us":23280,"threads_started":1}
I20260923 17:22:13.296003 3285 ts_manager.cc:194] Registered new tserver with Master: 230eab516f7b4c07b7ef07e40a37cd35 (127.28.72.195:46417)
I20260923 17:22:13.297060 3940 heartbeater.cc:347] Connected to a master server at 127.28.72.252:43907
I20260923 17:22:13.297214 3940 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:13.297883 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.252453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:43149 (local address 127.28.72.254:34367)
0923 17:22:13.275641 (+ 23188us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.275649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.276136 (+ 487us) server_negotiation.cc:407] Connection header received
0923 17:22:13.276390 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.276396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.276491 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.276638 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:13.278220 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.279351 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.281860 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.282556 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.283312 (+ 756us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.287420 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:13.287428 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:13.287434 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:13.287482 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:13.291267 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.294346 (+ 3079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.294356 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.294362 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.294527 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.296046 (+ 1519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.296054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.296060 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.296596 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:13.296758 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.297176 (+ 418us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.297460 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23039,"thread_start_us":22928,"threads_started":1}
I20260923 17:22:13.297793 3940 heartbeater.cc:511] Master 127.28.72.252:43907 requested a full tablet report, sending...
I20260923 17:22:13.297945 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.251941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.195:43149)
0923 17:22:13.276019 (+ 24078us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:13.276034 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:13.276230 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:13.276787 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:13.276791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:13.277189 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:13.277627 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.277640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.279546 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.279553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.281636 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.281657 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.282784 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.282790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.282988 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.283675 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:13.283698 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:13.287105 (+ 3407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:13.291401 (+ 4296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:13.291408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:13.291415 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:13.291972 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:13.294640 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:13.294645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:13.294648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:13.294814 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:13.296717 (+ 1903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:13.296723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:13.296892 (+ 169us) client_negotiation.cc:769] Sending connection context
0923 17:22:13.297050 (+ 158us) client_negotiation.cc:240] Negotiation successful
0923 17:22:13.297344 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23979,"thread_start_us":24402,"threads_started":1}
I20260923 17:22:13.298882 3941 heartbeater.cc:347] Connected to a master server at 127.28.72.254:34367
I20260923 17:22:13.299050 3941 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:13.299315 3475 ts_manager.cc:194] Registered new tserver with Master: 230eab516f7b4c07b7ef07e40a37cd35 (127.28.72.195:46417)
I20260923 17:22:13.299527 3941 heartbeater.cc:511] Master 127.28.72.254:34367 requested a full tablet report, sending...
I20260923 17:22:13.300522 2684 ts_manager.cc:194] Re-registered known tserver with Master: 230eab516f7b4c07b7ef07e40a37cd35 (127.28.72.195:46417)
I20260923 17:22:13.301947 2684 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:43149
I20260923 17:22:13.321187 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.292438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43134 (local address 127.28.72.253:33789)
0923 17:22:13.292635 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.292640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.301492 (+ 8852us) server_negotiation.cc:407] Connection header received
0923 17:22:13.301771 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.301777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.301874 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.301973 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:13.304879 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.305776 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.307474 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.308158 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.308269 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.312758 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:13.312769 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:13.312774 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:13.312820 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:13.316916 (+ 4096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.318022 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.318031 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.318038 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.318226 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.318656 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.318662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.318668 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.318979 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:13.319291 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.319691 (+ 400us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.319940 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:22:13.326581 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.288652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47802 (local address 127.28.72.254:34367)
0923 17:22:13.298118 (+ 9466us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.298124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.307547 (+ 9423us) server_negotiation.cc:407] Connection header received
0923 17:22:13.307764 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.307771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.307862 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.307968 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:13.311265 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.312609 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.314826 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.315559 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.315701 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.320170 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:13.320179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:13.320185 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:13.320234 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:13.323676 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.324397 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.324404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.324408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.324551 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.324975 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.324980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.324984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.325289 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:13.325583 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.325952 (+ 369us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.326182 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9349,"thread_start_us":21957,"threads_started":1}
I20260923 17:22:13.326880 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.299743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.28.72.252:43907)
0923 17:22:13.299925 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.299932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.308353 (+ 8421us) server_negotiation.cc:407] Connection header received
0923 17:22:13.308644 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.308650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.308774 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.308856 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:13.313217 (+ 4361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.314181 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.316589 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.317297 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.317457 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.320923 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:13.320932 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:13.320937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:13.320982 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:13.324138 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.324872 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.324876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.324878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.325009 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.325397 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.325402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.325404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.325674 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:13.325970 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.326302 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.326535 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:22:13.330928 2684 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:22:13.338812 2684 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:22:13.379154 3969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.359785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.28.72.195:46417)
0923 17:22:13.368415 (+ 8630us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.368422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.368453 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:22:13.368536 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.368543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.368761 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.368941 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:13.370118 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.372168 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.376284 (+ 4116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.377779 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.377938 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.378167 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.378545 (+ 378us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.378714 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8457,"thread_start_us":8266,"threads_started":1}
I20260923 17:22:13.379168 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.359516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:46417 (local address 127.0.0.1:49278)
0923 17:22:13.368001 (+ 8485us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:13.368031 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:13.368196 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:13.369134 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:13.369143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:13.369176 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:13.369663 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.369679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.372299 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.372308 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.374678 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.374693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.378012 (+ 3319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.378022 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.378306 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.378364 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 17:22:13.378494 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 17:22:13.378702 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8299,"thread_start_us":8119,"threads_started":1}
I20260923 17:22:13.382678 3971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.360214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34200 (local address 127.28.72.194:35325)
0923 17:22:13.373414 (+ 13200us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.373424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.373455 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:22:13.373556 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.373565 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.373841 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.374071 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:13.375491 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.376827 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.379075 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.380521 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.380657 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.380851 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.382196 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.382393 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13020,"thread_start_us":14602,"threads_started":1}
I20260923 17:22:13.382661 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.359935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:35325 (local address 127.0.0.1:34200)
0923 17:22:13.372786 (+ 12851us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:13.372805 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:13.372921 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:13.374092 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:13.374096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:13.374124 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:13.374729 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.374745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.376966 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.376971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.378949 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.378964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.380718 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.380724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.380906 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.380955 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 17:22:13.381062 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 17:22:13.381258 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12707,"thread_start_us":14291,"threads_started":1}
I20260923 17:22:13.383523 3875 tablet_service.cc:1467] Processing CreateTablet for tablet c614b77def7d4580999693add0893113 (DEFAULT_TABLE table=test-table [id=e608df3ef8684239a899549b1ccd4644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:13.385159 3875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c614b77def7d4580999693add0893113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:13.386754 3734 tablet_service.cc:1467] Processing CreateTablet for tablet c614b77def7d4580999693add0893113 (DEFAULT_TABLE table=test-table [id=e608df3ef8684239a899549b1ccd4644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:13.388019 3734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c614b77def7d4580999693add0893113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:13.388494 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.374678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:35277 (local address 127.0.0.1:59524)
0923 17:22:13.379440 (+ 4762us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:13.379469 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:13.379576 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:13.380308 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:13.380315 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:13.380341 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:13.380522 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.380532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.383077 (+ 2545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.383088 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.385050 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.385067 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.387655 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.387664 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.387861 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.387896 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:22:13.387990 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 17:22:13.388131 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4663,"thread_start_us":8666,"threads_started":1}
I20260923 17:22:13.389252 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.374951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.28.72.193:35277)
0923 17:22:13.376329 (+ 1378us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.376337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.379698 (+ 3361us) server_negotiation.cc:407] Connection header received
0923 17:22:13.379807 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.379817 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.380091 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.380299 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:13.380717 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.382568 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.385242 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.387164 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.387323 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.387541 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.388751 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.388947 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1208,"thread_start_us":1090,"threads_started":1}
I20260923 17:22:13.392859 3589 tablet_service.cc:1467] Processing CreateTablet for tablet c614b77def7d4580999693add0893113 (DEFAULT_TABLE table=test-table [id=e608df3ef8684239a899549b1ccd4644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:13.394326 3589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c614b77def7d4580999693add0893113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:13.399768 3974 tablet_bootstrap.cc:492] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35: Bootstrap starting.
I20260923 17:22:13.403255 3974 tablet_bootstrap.cc:654] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:13.404458 3975 tablet_bootstrap.cc:492] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f: Bootstrap starting.
I20260923 17:22:13.404908 3974 log.cc:824] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:13.408115 3975 tablet_bootstrap.cc:654] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:13.409201 3974 tablet_bootstrap.cc:492] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35: No bootstrap required, opened a new log
I20260923 17:22:13.409423 3975 log.cc:824] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:13.409734 3974 ts_tablet_manager.cc:1397] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260923 17:22:13.415263 3977 tablet_bootstrap.cc:492] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0: Bootstrap starting.
I20260923 17:22:13.416301 3975 tablet_bootstrap.cc:492] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f: No bootstrap required, opened a new log
I20260923 17:22:13.416597 3975 ts_tablet_manager.cc:1397] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260923 17:22:13.418133 3977 tablet_bootstrap.cc:654] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:13.419251 3977 log.cc:824] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:13.421940 3974 raft_consensus.cc:348] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } }
I20260923 17:22:13.422600 3974 raft_consensus.cc:374] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:13.422695 3974 raft_consensus.cc:729] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230eab516f7b4c07b7ef07e40a37cd35, State: Initialized, Role: FOLLOWER
I20260923 17:22:13.423475 3974 consensus_queue.cc:260] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } }
I20260923 17:22:13.428824 3975 raft_consensus.cc:348] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } }
I20260923 17:22:13.429514 3975 raft_consensus.cc:374] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:13.429658 3975 raft_consensus.cc:729] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19bba33d1f447cf94a81c91177d918f, State: Initialized, Role: FOLLOWER
I20260923 17:22:13.430490 3975 consensus_queue.cc:260] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } }
I20260923 17:22:13.430894 3941 heartbeater.cc:503] Master 127.28.72.254:34367 was elected leader, sending a full tablet report...
I20260923 17:22:13.431944 3974 ts_tablet_manager.cc:1428] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35: Time spent starting tablet: real 0.022s user 0.013s sys 0.008s
I20260923 17:22:13.433521 3975 ts_tablet_manager.cc:1428] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f: Time spent starting tablet: real 0.017s user 0.014s sys 0.003s
I20260923 17:22:13.433787 3800 heartbeater.cc:503] Master 127.28.72.254:34367 was elected leader, sending a full tablet report...
I20260923 17:22:13.442116 3977 tablet_bootstrap.cc:492] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0: No bootstrap required, opened a new log
I20260923 17:22:13.442461 3977 ts_tablet_manager.cc:1397] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.008s
I20260923 17:22:13.450276 3977 raft_consensus.cc:348] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } }
I20260923 17:22:13.450724 3977 raft_consensus.cc:374] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:13.450845 3977 raft_consensus.cc:729] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1d20be2ecb4a1891207977144917e0, State: Initialized, Role: FOLLOWER
I20260923 17:22:13.451517 3977 consensus_queue.cc:260] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } }
I20260923 17:22:13.453305 3977 ts_tablet_manager.cc:1428] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0: Time spent starting tablet: real 0.011s user 0.003s sys 0.008s
W20260923 17:22:13.519064 3943 tablet.cc:2369] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:22:13.584328 3802 tablet.cc:2369] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:22:13.594494 3980 raft_consensus.cc:482] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:13.594820 3980 raft_consensus.cc:504] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } }
I20260923 17:22:13.596792 3980 leader_election.cc:288] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d19bba33d1f447cf94a81c91177d918f (127.28.72.194:35325), 3b1d20be2ecb4a1891207977144917e0 (127.28.72.193:35277)
I20260923 17:22:13.605127 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.597981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:48911 (local address 127.28.72.193:35277)
0923 17:22:13.598175 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.598180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.598194 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:22:13.598247 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.598250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.598337 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.598432 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:13.598719 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.599764 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.602430 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.603779 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.603943 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.604022 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.604693 (+ 671us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.604869 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:22:13.605144 3971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.597164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:40593 (local address 127.28.72.194:35325)
0923 17:22:13.597347 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.597351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.597557 (+ 206us) server_negotiation.cc:407] Connection header received
0923 17:22:13.597784 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.597793 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.597877 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.597978 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:13.598387 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.599522 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.602450 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.603768 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.603943 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.604022 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.604694 (+ 672us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.604837 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:22:13.605255 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.597767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:35277 (local address 127.28.72.195:48911)
0923 17:22:13.597946 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:13.597966 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:13.598069 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:13.598460 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:13.598463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:13.598482 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:13.598595 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.598601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.599891 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.599896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.602242 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.602260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.604080 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.604093 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.604402 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.604511 (+ 109us) client_negotiation.cc:769] Sending connection context
0923 17:22:13.604664 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 17:22:13.604847 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":215,"spinlock_wait_cycles":7680}
I20260923 17:22:13.605276 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.597240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:35325 (local address 127.28.72.195:40593)
0923 17:22:13.597435 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:13.597456 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:13.597619 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:13.598011 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:13.598017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:13.598067 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:13.598250 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.598258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.599670 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.599674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.602242 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:13.602260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.604105 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.604110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:13.604399 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.604518 (+ 119us) client_negotiation.cc:769] Sending connection context
0923 17:22:13.604623 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 17:22:13.604846 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":219,"spinlock_wait_cycles":11520}
I20260923 17:22:13.606442 3609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c614b77def7d4580999693add0893113" candidate_uuid: "230eab516f7b4c07b7ef07e40a37cd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1d20be2ecb4a1891207977144917e0" is_pre_election: true
I20260923 17:22:13.606740 3754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c614b77def7d4580999693add0893113" candidate_uuid: "230eab516f7b4c07b7ef07e40a37cd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19bba33d1f447cf94a81c91177d918f" is_pre_election: true
I20260923 17:22:13.607007 3609 raft_consensus.cc:2454] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 230eab516f7b4c07b7ef07e40a37cd35 in term 0.
I20260923 17:22:13.607332 3754 raft_consensus.cc:2454] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 230eab516f7b4c07b7ef07e40a37cd35 in term 0.
I20260923 17:22:13.607945 3828 leader_election.cc:302] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 230eab516f7b4c07b7ef07e40a37cd35, 3b1d20be2ecb4a1891207977144917e0; no voters:
I20260923 17:22:13.608377 3980 raft_consensus.cc:2781] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:22:13.608592 3980 raft_consensus.cc:482] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:22:13.608709 3980 raft_consensus.cc:3037] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:13.611605 3980 raft_consensus.cc:504] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } }
I20260923 17:22:13.612227 3980 leader_election.cc:288] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [CANDIDATE]: Term 1 election: Requested vote from peers d19bba33d1f447cf94a81c91177d918f (127.28.72.194:35325), 3b1d20be2ecb4a1891207977144917e0 (127.28.72.193:35277)
I20260923 17:22:13.613029 3754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c614b77def7d4580999693add0893113" candidate_uuid: "230eab516f7b4c07b7ef07e40a37cd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19bba33d1f447cf94a81c91177d918f"
I20260923 17:22:13.613171 3609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c614b77def7d4580999693add0893113" candidate_uuid: "230eab516f7b4c07b7ef07e40a37cd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1d20be2ecb4a1891207977144917e0"
I20260923 17:22:13.613250 3754 raft_consensus.cc:3037] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:13.613507 3609 raft_consensus.cc:3037] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:13.615732 3754 raft_consensus.cc:2454] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 230eab516f7b4c07b7ef07e40a37cd35 in term 1.
I20260923 17:22:13.615924 3609 raft_consensus.cc:2454] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 230eab516f7b4c07b7ef07e40a37cd35 in term 1.
I20260923 17:22:13.616228 3828 leader_election.cc:302] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 230eab516f7b4c07b7ef07e40a37cd35, d19bba33d1f447cf94a81c91177d918f; no voters:
I20260923 17:22:13.616597 3980 raft_consensus.cc:2781] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:13.617072 3980 raft_consensus.cc:686] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [term 1 LEADER]: Becoming Leader. State: Replica: 230eab516f7b4c07b7ef07e40a37cd35, State: Running, Role: LEADER
I20260923 17:22:13.617693 3980 consensus_queue.cc:237] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } }
I20260923 17:22:13.623122 2683 catalog_manager.cc:5495] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 230eab516f7b4c07b7ef07e40a37cd35 (127.28.72.195). New cstate: current_term: 1 leader_uuid: "230eab516f7b4c07b7ef07e40a37cd35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } health_report { overall_health: UNKNOWN } } }
W20260923 17:22:13.659834 3657 tablet.cc:2369] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:22:13.667421 3969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.661851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.28.72.195:46417)
0923 17:22:13.662128 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.662135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.662155 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:22:13.662260 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.662264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.662343 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.662505 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:13.662888 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.663723 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.665197 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.665615 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.665755 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.666270 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:13.666797 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:13.666980 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.667027 (+ 47us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.667150 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:22:13.676684 3609 raft_consensus.cc:1261] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0 [term 1 FOLLOWER]: Refusing update from remote peer 230eab516f7b4c07b7ef07e40a37cd35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:13.676692 3754 raft_consensus.cc:1261] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f [term 1 FOLLOWER]: Refusing update from remote peer 230eab516f7b4c07b7ef07e40a37cd35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:13.677963 3983 consensus_queue.cc:1035] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:13.678404 3980 consensus_queue.cc:1035] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:13.719332 2683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:13.719658 2683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:13.726568 2683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9e1d06ed1444e00ae887cb39ee5a144: Sending DeleteTablet for 3 replicas of tablet c614b77def7d4580999693add0893113
I20260923 17:22:13.727813 3734 tablet_service.cc:1514] Processing DeleteTablet for tablet c614b77def7d4580999693add0893113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:13 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:34200
I20260923 17:22:13.728516 3875 tablet_service.cc:1514] Processing DeleteTablet for tablet c614b77def7d4580999693add0893113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:13 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:49278
I20260923 17:22:13.729081 3589 tablet_service.cc:1514] Processing DeleteTablet for tablet c614b77def7d4580999693add0893113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:13 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:59524
I20260923 17:22:13.729511 3996 tablet_replica.cc:330] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f: stopping tablet replica
I20260923 17:22:13.730089 3996 raft_consensus.cc:2229] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:13.738672 3996 raft_consensus.cc:2258] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:13.740587 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 2652
I20260923 17:22:13.740728 3997 tablet_replica.cc:330] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35: stopping tablet replica
I20260923 17:22:13.740911 3998 tablet_replica.cc:330] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0: stopping tablet replica
I20260923 17:22:13.741428 3997 raft_consensus.cc:2229] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:22:13.741428 3998 raft_consensus.cc:2229] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:13.741847 2714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:13.742012 3998 raft_consensus.cc:2258] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:13.742115 3997 raft_consensus.cc:2258] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:13.744387 3997 ts_tablet_manager.cc:1905] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:13.745285 3996 ts_tablet_manager.cc:1905] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:13.745711 3998 ts_tablet_manager.cc:1905] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:13.752744 3997 ts_tablet_manager.cc:1918] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:13.753034 3997 log.cc:1196] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/c614b77def7d4580999693add0893113
I20260923 17:22:13.753525 3998 ts_tablet_manager.cc:1918] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:13.753784 3998 log.cc:1196] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/c614b77def7d4580999693add0893113
I20260923 17:22:13.753772 3997 ts_tablet_manager.cc:1939] T c614b77def7d4580999693add0893113 P 230eab516f7b4c07b7ef07e40a37cd35: Deleting consensus metadata
I20260923 17:22:13.754411 3998 ts_tablet_manager.cc:1939] T c614b77def7d4580999693add0893113 P 3b1d20be2ecb4a1891207977144917e0: Deleting consensus metadata
I20260923 17:22:13.763463 3996 ts_tablet_manager.cc:1918] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:13.763795 3996 log.cc:1196] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/c614b77def7d4580999693add0893113
I20260923 17:22:13.764557 3996 ts_tablet_manager.cc:1939] T c614b77def7d4580999693add0893113 P d19bba33d1f447cf94a81c91177d918f: Deleting consensus metadata
I20260923 17:22:13.775024 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.731380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.28.72.253:33789)
0923 17:22:13.731556 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.731561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.739642 (+ 8081us) server_negotiation.cc:407] Connection header received
0923 17:22:13.739862 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.739868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.739959 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.740063 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:13.743565 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.744699 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.756515 (+ 11816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.757178 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.757326 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.767634 (+ 10308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:13.767642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:13.767647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:13.767695 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:13.771303 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.772306 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.772313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.772316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.772482 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.772970 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.772976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.772979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.773404 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:13.773688 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.774462 (+ 774us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.774681 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:22:13.793519 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.739875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50452 (local address 127.28.72.252:43907)
0923 17:22:13.740049 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.740055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.775084 (+ 35029us) server_negotiation.cc:407] Connection header received
0923 17:22:13.775299 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.775304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.775383 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.775496 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:13.778281 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.779566 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.781871 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.782576 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.782737 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.786505 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:13.786513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:13.786519 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:13.786565 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:13.790249 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.791253 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.791260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.791264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.791440 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.791893 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.791899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.791903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.792273 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:13.792543 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.792883 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.793134 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:22:13.892849 2668 catalog_manager.cc:4841] TS 3b1d20be2ecb4a1891207977144917e0 (127.28.72.193:35277): tablet c614b77def7d4580999693add0893113 (table test-table [id=e608df3ef8684239a899549b1ccd4644]) successfully deleted
W20260923 17:22:13.893339 2668 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.28.72.193:35277, user_credentials={real_user=slave}} blocked reactor thread for 136999us
I20260923 17:22:13.893935 2668 catalog_manager.cc:4841] TS d19bba33d1f447cf94a81c91177d918f (127.28.72.194:35325): tablet c614b77def7d4580999693add0893113 (table test-table [id=e608df3ef8684239a899549b1ccd4644]) successfully deleted
I20260923 17:22:13.894058 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 2652
I20260923 17:22:13.948853 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.922048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43158 (local address 127.28.72.253:33789)
0923 17:22:13.922259 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.922267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.930138 (+ 7871us) server_negotiation.cc:407] Connection header received
0923 17:22:13.930421 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.930427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.930511 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.930578 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:13.934242 (+ 3664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.935402 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.936881 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.937541 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.937680 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.941495 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:13.941520 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:13.941525 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:13.941570 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:13.944817 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.946448 (+ 1631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.946457 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.946461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.946645 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.947082 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.947088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.947091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.947580 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:13.947838 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.948248 (+ 410us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.948480 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:22:13.950374 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.928543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.28.72.252:43907)
0923 17:22:13.928812 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:13.928819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:13.934100 (+ 5281us) server_negotiation.cc:407] Connection header received
0923 17:22:13.934447 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:13.934452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:13.934537 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:13.934662 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:13.937039 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.938024 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.939924 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:13.940366 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:13.940509 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:13.943979 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:13.943986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:13.943990 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:13.944029 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:13.947270 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.948053 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.948061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.948067 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.948219 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:13.948801 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:13.948808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:13.948816 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:13.949191 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:13.949469 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:13.949793 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 17:22:13.950035 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 17:22:14.702322 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:14.701474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.194:44815)
0923 17:22:14.701952 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:14.702054 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":249,"threads_started":1}
W20260923 17:22:14.702860 3800 heartbeater.cc:644] Failed to heartbeat to 127.28.72.254:34367 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
I20260923 17:22:14.711490 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:14.710531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.193:58951)
0923 17:22:14.711020 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:14.711198 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":288,"threads_started":1}
W20260923 17:22:14.711918 3655 heartbeater.cc:644] Failed to heartbeat to 127.28.72.254:34367 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
I20260923 17:22:14.715937 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:14.714992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.195:35935)
0923 17:22:14.715518 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:14.715680 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":301,"threads_started":1}
W20260923 17:22:14.716320 3941 heartbeater.cc:644] Failed to heartbeat to 127.28.72.254:34367 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
I20260923 17:22:15.227252 4011 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a9e1d06ed1444e00ae887cb39ee5a144)
I20260923 17:22:15.227603 4011 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } }
I20260923 17:22:15.227733 4012 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a9e1d06ed1444e00ae887cb39ee5a144)
I20260923 17:22:15.227969 4012 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } }
I20260923 17:22:15.229162 4011 leader_election.cc:288] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9e1d06ed1444e00ae887cb39ee5a144 (127.28.72.254:34367), d1ad5c22fa4a47119df88053feb844f4 (127.28.72.253:33789)
I20260923 17:22:15.230433 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.229301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:48256)
0923 17:22:15.229902 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.230136 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":305,"threads_started":1}
I20260923 17:22:15.230670 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.229283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:48268)
0923 17:22:15.230207 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.230421 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":661,"threads_started":1}
W20260923 17:22:15.230823 3459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
W20260923 17:22:15.231334 3270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
I20260923 17:22:15.241139 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.229317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:33789 (local address 127.0.0.1:43160)
0923 17:22:15.231664 (+ 2347us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.231687 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:15.231893 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:15.232618 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:15.232626 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:15.232662 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:15.232963 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:15.232980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.235204 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.235214 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:15.238138 (+ 2924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:15.238155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.240164 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.240174 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:15.240471 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:15.240510 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 17:22:15.240611 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 17:22:15.240773 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2193,"threads_started":1}
I20260923 17:22:15.241245 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.230249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.28.72.253:33789)
0923 17:22:15.231398 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:15.231408 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:15.232044 (+ 636us) server_negotiation.cc:407] Connection header received
0923 17:22:15.232140 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:15.232148 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:15.232257 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:15.232378 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:15.233232 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.234936 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.238368 (+ 3432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.239689 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.239865 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:15.240033 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:15.240697 (+ 664us) server_negotiation.cc:299] Negotiation successful
0923 17:22:15.240855 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":751,"threads_started":1}
I20260923 17:22:15.241742 4012 leader_election.cc:288] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9e1d06ed1444e00ae887cb39ee5a144 (127.28.72.254:34367), c3b8453220e145eeaba2927bf0cc938c (127.28.72.252:43907)
I20260923 17:22:15.242430 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.241975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:48288)
0923 17:22:15.242158 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.242256 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 17:22:15.242420 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.241811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:48286)
0923 17:22:15.242112 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.242212 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260923 17:22:15.242671 3459 leader_election.cc:334] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a9e1d06ed1444e00ae887cb39ee5a144 (127.28.72.254:34367): Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
W20260923 17:22:15.243078 3270 leader_election.cc:334] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a9e1d06ed1444e00ae887cb39ee5a144 (127.28.72.254:34367): Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
I20260923 17:22:15.243736 3295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3b8453220e145eeaba2927bf0cc938c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d1ad5c22fa4a47119df88053feb844f4" is_pre_election: true
I20260923 17:22:15.244073 3295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b8453220e145eeaba2927bf0cc938c in term 1.
I20260923 17:22:15.244665 3460 leader_election.cc:302] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c3b8453220e145eeaba2927bf0cc938c, d1ad5c22fa4a47119df88053feb844f4; no voters: a9e1d06ed1444e00ae887cb39ee5a144
I20260923 17:22:15.245090 4011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 17:22:15.245223 4011 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a9e1d06ed1444e00ae887cb39ee5a144)
I20260923 17:22:15.245340 4011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:22:15.247562 4011 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } }
I20260923 17:22:15.248268 4011 leader_election.cc:288] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [CANDIDATE]: Term 2 election: Requested vote from peers a9e1d06ed1444e00ae887cb39ee5a144 (127.28.72.254:34367), d1ad5c22fa4a47119df88053feb844f4 (127.28.72.253:33789)
I20260923 17:22:15.248819 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.248349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:48304)
0923 17:22:15.248553 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.248638 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260923 17:22:15.248840 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.242586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43907 (local address 127.0.0.1:50476)
0923 17:22:15.242726 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.242746 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:15.242871 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:15.243755 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:15.243760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:15.243793 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:15.244118 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:15.244129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.245407 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.245416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:15.246727 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:15.246738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.248032 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.248038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:15.248215 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:15.248279 (+ 64us) client_negotiation.cc:769] Sending connection context
0923 17:22:15.248373 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 17:22:15.248523 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 17:22:15.248744 3295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3b8453220e145eeaba2927bf0cc938c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d1ad5c22fa4a47119df88053feb844f4"
I20260923 17:22:15.249028 3295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:22:15.249454 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.242847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.28.72.252:43907)
0923 17:22:15.243456 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:15.243462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:15.243485 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:22:15.243540 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:15.243545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:15.243615 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:15.243740 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:15.244307 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.245286 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.246894 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.247845 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.247991 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:15.248100 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:15.248525 (+ 425us) server_negotiation.cc:299] Negotiation successful
0923 17:22:15.248656 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":376,"threads_started":1}
I20260923 17:22:15.249909 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.249461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:48308)
0923 17:22:15.249650 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.249733 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260923 17:22:15.250103 3459 leader_election.cc:334] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a9e1d06ed1444e00ae887cb39ee5a144 (127.28.72.254:34367): Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
I20260923 17:22:15.250123 3485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1ad5c22fa4a47119df88053feb844f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c3b8453220e145eeaba2927bf0cc938c" is_pre_election: true
I20260923 17:22:15.250517 3485 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1ad5c22fa4a47119df88053feb844f4 in current term 2: Already voted for candidate c3b8453220e145eeaba2927bf0cc938c in this term.
I20260923 17:22:15.251261 3272 leader_election.cc:302] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1ad5c22fa4a47119df88053feb844f4; no voters: a9e1d06ed1444e00ae887cb39ee5a144, c3b8453220e145eeaba2927bf0cc938c
I20260923 17:22:15.251899 3295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b8453220e145eeaba2927bf0cc938c in term 2.
I20260923 17:22:15.252077 4012 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:22:15.252328 3460 leader_election.cc:302] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c3b8453220e145eeaba2927bf0cc938c, d1ad5c22fa4a47119df88053feb844f4; no voters: a9e1d06ed1444e00ae887cb39ee5a144
I20260923 17:22:15.252722 4011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:22:15.253180 4011 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [term 2 LEADER]: Becoming Leader. State: Replica: c3b8453220e145eeaba2927bf0cc938c, State: Running, Role: LEADER
I20260923 17:22:15.253624 4011 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } }
I20260923 17:22:15.256076 4020 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3b8453220e145eeaba2927bf0cc938c. Latest consensus state: current_term: 2 leader_uuid: "c3b8453220e145eeaba2927bf0cc938c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } } }
I20260923 17:22:15.256361 4020 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:15.257052 4022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:15.260746 4022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:15.261731 4022 catalog_manager.cc:1259] Loaded cluster ID: fe89399da8a04b4db57b7b77d918fd64
I20260923 17:22:15.261803 4022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:15.262941 4022 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:15.263845 4022 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: Loaded TSK: 0
I20260923 17:22:15.264750 4022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:22:15.308768 3474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:22:15.315891 3474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:22:15.320449 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.320024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:48320)
0923 17:22:15.320167 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.320282 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 17:22:15.320601 3295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [term 2 FOLLOWER]: Refusing update from remote peer c3b8453220e145eeaba2927bf0cc938c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260923 17:22:15.321532 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.321123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:48332)
0923 17:22:15.321278 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.321391 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 17:22:15.321488 4020 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260923 17:22:15.321785 3459 consensus_peers.cc:487] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c -> Peer a9e1d06ed1444e00ae887cb39ee5a144 (127.28.72.254:34367): Couldn't send request to peer a9e1d06ed1444e00ae887cb39ee5a144. Status: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 17:22:15.324952 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3b8453220e145eeaba2927bf0cc938c. Latest consensus state: current_term: 2 leader_uuid: "c3b8453220e145eeaba2927bf0cc938c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } } }
I20260923 17:22:15.325213 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:22:15.326289 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c3b8453220e145eeaba2927bf0cc938c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } } }
I20260923 17:22:15.326704 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:15.331386 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c3b8453220e145eeaba2927bf0cc938c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a9e1d06ed1444e00ae887cb39ee5a144" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 34367 } } peers { permanent_uuid: "d1ad5c22fa4a47119df88053feb844f4" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 33789 } attrs { promote: false } } peers { permanent_uuid: "c3b8453220e145eeaba2927bf0cc938c" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43907 } attrs { promote: false } } }
I20260923 17:22:15.331668 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ad5c22fa4a47119df88053feb844f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:22:15.348335 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.336879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:35325 (local address 127.0.0.1:34206)
0923 17:22:15.337045 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.337061 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:15.337154 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:15.338110 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:15.338118 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:15.338154 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:15.338697 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:15.338713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.340887 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.340897 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:15.344018 (+ 3121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:15.344053 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.347105 (+ 3052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.347134 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:15.347427 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:15.347504 (+ 77us) client_negotiation.cc:769] Sending connection context
0923 17:22:15.347603 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 17:22:15.347827 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 17:22:15.350718 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.337215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.28.72.194:35325)
0923 17:22:15.337718 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:15.337727 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:15.337745 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:22:15.337815 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:15.337821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:15.337916 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:15.338921 (+ 1005us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:15.339157 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.340655 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.344968 (+ 4313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.346741 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.349778 (+ 3037us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:15.349946 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:15.350023 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 17:22:15.350249 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":691,"threads_started":1}
I20260923 17:22:15.352089 3734 tablet_service.cc:1467] Processing CreateTablet for tablet b2b1e98ec6b84c73b7a65fbbff75605e (DEFAULT_TABLE table=test-table [id=4214404e07a04c769461d2238c7091bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:15.352746 3734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b1e98ec6b84c73b7a65fbbff75605e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:15.359292 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.337673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:46417 (local address 127.0.0.1:49284)
0923 17:22:15.349331 (+ 11658us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.349359 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:15.349530 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:15.353737 (+ 4207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:15.353742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:15.353772 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:15.354000 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:15.354011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.355554 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.355560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:15.357177 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:15.357188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.358459 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.358465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:15.358682 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:15.358716 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:22:15.358785 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 17:22:15.358931 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11463,"thread_start_us":11705,"threads_started":1}
I20260923 17:22:15.359480 4033 tablet_bootstrap.cc:492] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f: Bootstrap starting.
I20260923 17:22:15.360087 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.336572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59536 (local address 127.28.72.193:35277)
0923 17:22:15.352288 (+ 15716us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:15.352293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:15.352337 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:22:15.352416 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:15.352422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:15.352504 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:15.352636 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:15.353051 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.354110 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.356556 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.357805 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.358538 (+ 733us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:15.358664 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:15.359683 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0923 17:22:15.359835 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15612,"thread_start_us":16181,"threads_started":1}
I20260923 17:22:15.360080 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.336385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:35277 (local address 127.0.0.1:59536)
0923 17:22:15.336659 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.336679 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:15.336786 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:15.352663 (+ 15877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:15.352667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:15.352701 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:15.352879 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:15.352899 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.354260 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.354268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:15.356350 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:15.356368 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.358298 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.358305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:15.359500 (+ 1195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:15.359542 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 17:22:15.359637 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 17:22:15.359785 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":986}
I20260923 17:22:15.361078 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.337763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.28.72.195:46417)
0923 17:22:15.353133 (+ 15370us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:15.353139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:15.353159 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:22:15.353226 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:15.353234 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:15.353311 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:15.353425 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:15.354200 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.355366 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.357353 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.358265 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.360467 (+ 2202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:15.360592 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:15.360653 (+ 61us) server_negotiation.cc:299] Negotiation successful
0923 17:22:15.360780 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15261,"thread_start_us":15098,"threads_started":1}
I20260923 17:22:15.361294 3589 tablet_service.cc:1467] Processing CreateTablet for tablet b2b1e98ec6b84c73b7a65fbbff75605e (DEFAULT_TABLE table=test-table [id=4214404e07a04c769461d2238c7091bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:15.361750 3589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b1e98ec6b84c73b7a65fbbff75605e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:15.361886 3875 tablet_service.cc:1467] Processing CreateTablet for tablet b2b1e98ec6b84c73b7a65fbbff75605e (DEFAULT_TABLE table=test-table [id=4214404e07a04c769461d2238c7091bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:15.362284 4033 tablet_bootstrap.cc:654] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:15.362291 3875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b1e98ec6b84c73b7a65fbbff75605e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:15.364405 4033 tablet_bootstrap.cc:492] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f: No bootstrap required, opened a new log
I20260923 17:22:15.364576 4033 ts_tablet_manager.cc:1397] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260923 17:22:15.365253 4033 raft_consensus.cc:348] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } }
I20260923 17:22:15.365484 4033 raft_consensus.cc:374] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:15.365630 4033 raft_consensus.cc:729] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19bba33d1f447cf94a81c91177d918f, State: Initialized, Role: FOLLOWER
I20260923 17:22:15.365950 4033 consensus_queue.cc:260] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } }
I20260923 17:22:15.366935 4033 ts_tablet_manager.cc:1428] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260923 17:22:15.368637 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.367544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.194:48187)
0923 17:22:15.368177 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.368357 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":340,"threads_started":1}
I20260923 17:22:15.369309 4037 tablet_bootstrap.cc:492] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35: Bootstrap starting.
I20260923 17:22:15.370386 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.370070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.194:48717)
0923 17:22:15.370205 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.370282 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 17:22:15.371296 3799 heartbeater.cc:503] Master 127.28.72.252:43907 was elected leader, sending a full tablet report...
I20260923 17:22:15.372594 4037 tablet_bootstrap.cc:654] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:15.373953 4038 tablet_bootstrap.cc:492] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0: Bootstrap starting.
I20260923 17:22:15.375401 4037 tablet_bootstrap.cc:492] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35: No bootstrap required, opened a new log
I20260923 17:22:15.375558 4037 ts_tablet_manager.cc:1397] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:22:15.375864 4038 tablet_bootstrap.cc:654] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:15.376318 4037 raft_consensus.cc:348] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } }
I20260923 17:22:15.376505 4037 raft_consensus.cc:374] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:15.376607 4037 raft_consensus.cc:729] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230eab516f7b4c07b7ef07e40a37cd35, State: Initialized, Role: FOLLOWER
I20260923 17:22:15.376892 4037 consensus_queue.cc:260] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } }
I20260923 17:22:15.377784 4037 ts_tablet_manager.cc:1428] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:22:15.379024 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.378038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.195:45443)
0923 17:22:15.378583 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.378762 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":550,"threads_started":1}
I20260923 17:22:15.380121 4038 tablet_bootstrap.cc:492] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0: No bootstrap required, opened a new log
I20260923 17:22:15.380214 3940 heartbeater.cc:503] Master 127.28.72.252:43907 was elected leader, sending a full tablet report...
I20260923 17:22:15.380294 4038 ts_tablet_manager.cc:1397] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:22:15.381138 4038 raft_consensus.cc:348] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } }
I20260923 17:22:15.381386 4038 raft_consensus.cc:374] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:15.381534 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.379800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.195:52865)
0923 17:22:15.381174 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.381348 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1265}
I20260923 17:22:15.381505 4038 raft_consensus.cc:729] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1d20be2ecb4a1891207977144917e0, State: Initialized, Role: FOLLOWER
I20260923 17:22:15.381807 4038 consensus_queue.cc:260] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } }
I20260923 17:22:15.382514 4038 ts_tablet_manager.cc:1428] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:22:15.384450 3654 heartbeater.cc:503] Master 127.28.72.252:43907 was elected leader, sending a full tablet report...
I20260923 17:22:15.384730 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.382765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.193:60271)
0923 17:22:15.384228 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.384432 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":296,"threads_started":1}
I20260923 17:22:15.385950 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.385625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.193:43585)
0923 17:22:15.385765 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.385841 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 17:22:15.665078 4041 raft_consensus.cc:482] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:15.665344 4041 raft_consensus.cc:504] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } }
I20260923 17:22:15.666103 4041 leader_election.cc:288] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b1d20be2ecb4a1891207977144917e0 (127.28.72.193:35277), d19bba33d1f447cf94a81c91177d918f (127.28.72.194:35325)
I20260923 17:22:15.666769 3609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b1e98ec6b84c73b7a65fbbff75605e" candidate_uuid: "230eab516f7b4c07b7ef07e40a37cd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1d20be2ecb4a1891207977144917e0" is_pre_election: true
I20260923 17:22:15.667044 3609 raft_consensus.cc:2454] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 230eab516f7b4c07b7ef07e40a37cd35 in term 0.
I20260923 17:22:15.666927 3754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b1e98ec6b84c73b7a65fbbff75605e" candidate_uuid: "230eab516f7b4c07b7ef07e40a37cd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19bba33d1f447cf94a81c91177d918f" is_pre_election: true
I20260923 17:22:15.667316 3754 raft_consensus.cc:2454] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 230eab516f7b4c07b7ef07e40a37cd35 in term 0.
I20260923 17:22:15.667649 3828 leader_election.cc:302] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 230eab516f7b4c07b7ef07e40a37cd35, 3b1d20be2ecb4a1891207977144917e0; no voters:
I20260923 17:22:15.667959 4041 raft_consensus.cc:2781] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:22:15.668084 4041 raft_consensus.cc:482] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:22:15.668180 4041 raft_consensus.cc:3037] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:15.670338 4041 raft_consensus.cc:504] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } }
I20260923 17:22:15.670890 4041 leader_election.cc:288] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [CANDIDATE]: Term 1 election: Requested vote from peers 3b1d20be2ecb4a1891207977144917e0 (127.28.72.193:35277), d19bba33d1f447cf94a81c91177d918f (127.28.72.194:35325)
I20260923 17:22:15.671605 3754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b1e98ec6b84c73b7a65fbbff75605e" candidate_uuid: "230eab516f7b4c07b7ef07e40a37cd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19bba33d1f447cf94a81c91177d918f"
I20260923 17:22:15.671625 3609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b1e98ec6b84c73b7a65fbbff75605e" candidate_uuid: "230eab516f7b4c07b7ef07e40a37cd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1d20be2ecb4a1891207977144917e0"
I20260923 17:22:15.671825 3754 raft_consensus.cc:3037] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:15.671841 3609 raft_consensus.cc:3037] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:15.674063 3609 raft_consensus.cc:2454] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 230eab516f7b4c07b7ef07e40a37cd35 in term 1.
I20260923 17:22:15.674111 3754 raft_consensus.cc:2454] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 230eab516f7b4c07b7ef07e40a37cd35 in term 1.
I20260923 17:22:15.674603 3828 leader_election.cc:302] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 230eab516f7b4c07b7ef07e40a37cd35, 3b1d20be2ecb4a1891207977144917e0; no voters:
I20260923 17:22:15.674957 4041 raft_consensus.cc:2781] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:15.675427 4041 raft_consensus.cc:686] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [term 1 LEADER]: Becoming Leader. State: Replica: 230eab516f7b4c07b7ef07e40a37cd35, State: Running, Role: LEADER
I20260923 17:22:15.675731 4041 consensus_queue.cc:237] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } } peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } }
I20260923 17:22:15.677016 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.676238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.195:49637)
0923 17:22:15.676388 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.676502 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 17:22:15.678784 3474 catalog_manager.cc:5495] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 230eab516f7b4c07b7ef07e40a37cd35 (127.28.72.195). New cstate: current_term: 1 leader_uuid: "230eab516f7b4c07b7ef07e40a37cd35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "230eab516f7b4c07b7ef07e40a37cd35" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 46417 } health_report { overall_health: HEALTHY } } }
I20260923 17:22:15.734884 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.730286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.28.72.195:46417)
0923 17:22:15.730599 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:15.730603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:15.730621 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:22:15.730677 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:15.730680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:15.730743 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:15.730827 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:15.731281 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.731992 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.733272 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.733677 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.733769 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:15.734154 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:15.734254 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:15.734331 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:15.734478 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 17:22:15.734556 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260923 17:22:15.738113 3609 raft_consensus.cc:1261] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0 [term 1 FOLLOWER]: Refusing update from remote peer 230eab516f7b4c07b7ef07e40a37cd35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:15.738159 3754 raft_consensus.cc:1261] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f [term 1 FOLLOWER]: Refusing update from remote peer 230eab516f7b4c07b7ef07e40a37cd35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:15.738785 4045 consensus_queue.cc:1035] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19bba33d1f447cf94a81c91177d918f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 35325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:15.739142 4041 consensus_queue.cc:1035] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1d20be2ecb4a1891207977144917e0" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 35277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:15.742120 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.741356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.194:53103)
0923 17:22:15.741699 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.741842 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260923 17:22:15.743820 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.743087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.193:53325)
0923 17:22:15.743351 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.743535 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 17:22:15.747313 4048 mvcc.cc:205] Tried to move back new op lower bound from 7332594219977207808 to 7332594219730059264. Current Snapshot: MvccSnapshot[applied={T|T < 7332594219977207808}]
I20260923 17:22:15.747951 4049 mvcc.cc:205] Tried to move back new op lower bound from 7332594219977207808 to 7332594219730059264. Current Snapshot: MvccSnapshot[applied={T|T < 7332594219977207808}]
I20260923 17:22:15.751636 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.748913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.193:32975)
0923 17:22:15.751356 (+ 2443us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.751467 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2317}
I20260923 17:22:15.754411 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.753891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.195:41247)
0923 17:22:15.754106 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.754207 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260923 17:22:15.755453 3474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:15.755702 3474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:15.757352 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.756885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.28.72.194:40767)
0923 17:22:15.757106 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.757199 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260923 17:22:15.761992 3474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3b8453220e145eeaba2927bf0cc938c: Sending DeleteTablet for 3 replicas of tablet b2b1e98ec6b84c73b7a65fbbff75605e
I20260923 17:22:15.762991 3734 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b1e98ec6b84c73b7a65fbbff75605e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:15 UTC) from {username='oryx', principal='oryx/127.28.72.252@KRBTEST.COM'} at 127.0.0.1:34206
I20260923 17:22:15.763139 3589 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b1e98ec6b84c73b7a65fbbff75605e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:15 UTC) from {username='oryx', principal='oryx/127.28.72.252@KRBTEST.COM'} at 127.0.0.1:59536
I20260923 17:22:15.763176 3733 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b1e98ec6b84c73b7a65fbbff75605e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:15 UTC) from {username='oryx', principal='oryx/127.28.72.252@KRBTEST.COM'} at 127.0.0.1:34206
I20260923 17:22:15.763527 3875 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b1e98ec6b84c73b7a65fbbff75605e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:15 UTC) from {username='oryx', principal='oryx/127.28.72.252@KRBTEST.COM'} at 127.0.0.1:49284
I20260923 17:22:15.763859 4057 tablet_replica.cc:330] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0: stopping tablet replica
I20260923 17:22:15.764189 4057 raft_consensus.cc:2229] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:15.764298 4058 tablet_replica.cc:330] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35: stopping tablet replica
I20260923 17:22:15.764546 4058 raft_consensus.cc:2229] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:22:15.764634 4057 raft_consensus.cc:2258] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:15.765280 4058 raft_consensus.cc:2258] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:15.765354 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 3465
I20260923 17:22:15.768471 4058 ts_tablet_manager.cc:1905] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:15.769294 4057 ts_tablet_manager.cc:1905] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:15.775422 4056 tablet_replica.cc:330] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f: stopping tablet replica
I20260923 17:22:15.775655 4056 raft_consensus.cc:2229] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:15.776114 4056 raft_consensus.cc:2258] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:15.777274 4056 ts_tablet_manager.cc:1905] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:15.777401 4058 ts_tablet_manager.cc:1918] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:15.777415 4057 ts_tablet_manager.cc:1918] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:15.777544 4058 log.cc:1196] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/b2b1e98ec6b84c73b7a65fbbff75605e
I20260923 17:22:15.777556 4057 log.cc:1196] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/b2b1e98ec6b84c73b7a65fbbff75605e
I20260923 17:22:15.778049 4058 ts_tablet_manager.cc:1939] T b2b1e98ec6b84c73b7a65fbbff75605e P 230eab516f7b4c07b7ef07e40a37cd35: Deleting consensus metadata
I20260923 17:22:15.778049 4057 ts_tablet_manager.cc:1939] T b2b1e98ec6b84c73b7a65fbbff75605e P 3b1d20be2ecb4a1891207977144917e0: Deleting consensus metadata
I20260923 17:22:15.779357 3461 catalog_manager.cc:4841] TS 230eab516f7b4c07b7ef07e40a37cd35 (127.28.72.195:46417): tablet b2b1e98ec6b84c73b7a65fbbff75605e (table test-table [id=4214404e07a04c769461d2238c7091bd]) successfully deleted
I20260923 17:22:15.779724 3458 catalog_manager.cc:4841] TS 3b1d20be2ecb4a1891207977144917e0 (127.28.72.193:35277): tablet b2b1e98ec6b84c73b7a65fbbff75605e (table test-table [id=4214404e07a04c769461d2238c7091bd]) successfully deleted
I20260923 17:22:15.782748 4056 ts_tablet_manager.cc:1918] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:15.782862 4056 log.cc:1196] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/b2b1e98ec6b84c73b7a65fbbff75605e
I20260923 17:22:15.783300 4056 ts_tablet_manager.cc:1939] T b2b1e98ec6b84c73b7a65fbbff75605e P d19bba33d1f447cf94a81c91177d918f: Deleting consensus metadata
I20260923 17:22:15.784497 3458 catalog_manager.cc:4841] TS d19bba33d1f447cf94a81c91177d918f (127.28.72.194:35325): tablet b2b1e98ec6b84c73b7a65fbbff75605e (table test-table [id=4214404e07a04c769461d2238c7091bd]) successfully deleted
W20260923 17:22:15.785104 3458 catalog_manager.cc:4808] TS d19bba33d1f447cf94a81c91177d918f (127.28.72.194:35325): delete failed for tablet b2b1e98ec6b84c73b7a65fbbff75605e because the tablet was not found. No further retry: Not found: Tablet not found: b2b1e98ec6b84c73b7a65fbbff75605e
I20260923 17:22:15.786353 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.766675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58896 (local address 127.28.72.193:35277)
0923 17:22:15.766954 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:15.766962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:15.768058 (+ 1096us) server_negotiation.cc:407] Connection header received
0923 17:22:15.768359 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:15.768366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:15.768449 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:15.768540 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:15.770877 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.772141 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.774099 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.774810 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.774930 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:15.779528 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:15.779550 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:15.779572 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:15.779613 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:15.782655 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:15.783748 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:15.783758 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:15.783761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:15.783966 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:15.784450 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:15.784455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:15.784458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:15.784956 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:15.785302 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:15.785800 (+ 498us) server_negotiation.cc:299] Negotiation successful
0923 17:22:15.785997 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 17:22:15.787036 3649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:15.881706 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.880976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:56440)
0923 17:22:15.881446 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.881539 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":245,"threads_started":1}
I20260923 17:22:15.883066 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.882675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:34367 (local address 127.0.0.1:56450)
0923 17:22:15.882868 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.882942 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 17:22:15.885504 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 3465
I20260923 17:22:15.906832 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 3666
I20260923 17:22:15.920872 4063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.907639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39470 (local address 127.28.72.194:35325)
0923 17:22:15.908164 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:15.908168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:15.908185 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:22:15.908238 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:15.908241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:15.908304 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:15.908393 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:15.909729 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.910452 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.911864 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:15.912252 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:15.912353 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:15.914989 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:15.915004 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:15.915020 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:15.915068 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:15.918032 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:15.918790 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:15.918798 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:15.918800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:15.918943 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:15.919393 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:15.919397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:15.919400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:15.919776 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:15.920005 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:15.920258 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 17:22:15.920497 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":245,"threads_started":1}
I20260923 17:22:15.921527 3794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:16.017875 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 3666
I20260923 17:22:16.036084 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 3812
I20260923 17:22:16.052439 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:16.036837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.28.72.195:46417)
0923 17:22:16.037022 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:16.037029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:16.037105 (+ 76us) server_negotiation.cc:407] Connection header received
0923 17:22:16.037316 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:16.037320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:16.037398 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:16.037568 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:16.038893 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:16.039668 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:16.040945 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:16.041339 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:16.041487 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:16.045165 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:16.045185 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:16.045201 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:16.045243 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:16.049190 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:16.050073 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:16.050081 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:16.050083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:16.050244 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:16.050731 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:16.050735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:16.050737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:16.051287 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:16.051556 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:16.051911 (+ 355us) server_negotiation.cc:299] Negotiation successful
0923 17:22:16.052159 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:22:16.053363 3935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:16.174737 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 3812
I20260923 17:22:16.198455 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 3147
I20260923 17:22:16.199836 3315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:16.320659 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 3147
I20260923 17:22:16.342097 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 3249
I20260923 17:22:16.343386 3505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:16.464787 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 3249
2026-09-23T17:22:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10364 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:22:16 dist-test-slave-j0m4 krb5kdc[4071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:22:16 dist-test-slave-j0m4 krb5kdc[4071](info): set up 2 sockets
Sep 23 17:22:16 dist-test-slave-j0m4 krb5kdc[4071](info): commencing operation
krb5kdc: starting...
W20260923 17:22:17.572628 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:22:17 dist-test-slave-j0m4 krb5kdc[4071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:22:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:22:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.254.keytab.
Entry for principal oryx/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.254.keytab.
I20260923 17:22:17.808091 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:38211
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.254.keytab
--principal=oryx/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:38211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:18.148268 4084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:18.148588 4084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:18.148725 4084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:18.156077 4084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:18.156204 4084 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:22:18.156280 4084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:18.156383 4084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:18.156466 4084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:18.156580 4084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:18.166798 4084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:38211
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.254.keytab
--principal=oryx/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:38211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:18.168248 4084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:18.169653 4084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:18.182574 4089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:18.182631 4090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:18.182633 4092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:18.183928 4084 server_base.cc:1034] running on GCE node
I20260923 17:22:18.184942 4084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:18.186514 4084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:18.187783 4084 hybrid_clock.cc:648] HybridClock initialized: now 1790184138187712 us; error 50 us; skew 500 ppm
I20260923 17:22:18.188172 4084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:18 dist-test-slave-j0m4 krb5kdc[4071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184138, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:18.191859 4084 init.cc:377] Logged in from keytab as oryx/127.28.72.254@KRBTEST.COM (short username oryx)
I20260923 17:22:18.193615 4084 webserver.cc:460] Webserver started at http://127.28.72.254:46313/ using document root <none> and password file <none>
I20260923 17:22:18.194437 4084 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:18.194581 4084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:18.194926 4084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:18.198208 4084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "e584ffce34a34c61b58a05e30d566379"
format_stamp: "Formatted at 2026-09-23 17:22:18 on dist-test-slave-j0m4"
server_key: "f536821e2016b703e07d8eec61801dd9"
server_key_iv: "da03879cec416a90f7f2cb33ed605e59"
server_key_version: "encryptionkey@0"
I20260923 17:22:18.199059 4084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "e584ffce34a34c61b58a05e30d566379"
format_stamp: "Formatted at 2026-09-23 17:22:18 on dist-test-slave-j0m4"
server_key: "f536821e2016b703e07d8eec61801dd9"
server_key_iv: "da03879cec416a90f7f2cb33ed605e59"
server_key_version: "encryptionkey@0"
I20260923 17:22:18.204912 4084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:22:18.208981 4099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:18.210664 4084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:22:18.210882 4084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "e584ffce34a34c61b58a05e30d566379"
format_stamp: "Formatted at 2026-09-23 17:22:18 on dist-test-slave-j0m4"
server_key: "f536821e2016b703e07d8eec61801dd9"
server_key_iv: "da03879cec416a90f7f2cb33ed605e59"
server_key_version: "encryptionkey@0"
I20260923 17:22:18.211176 4084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:18.241050 4084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:18.245467 4084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:18.245757 4084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:18.268189 4084 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:38211
I20260923 17:22:18.268208 4160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:38211 every 8 connection(s)
I20260923 17:22:18.269675 4084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:18.271378 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 4084
I20260923 17:22:18.271617 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:22:18.272408 28963 external_mini_cluster.cc:1428] Setting key df1ca8340a3c9d29ca57a4c64baa37f3
I20260923 17:22:18.274885 4161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:22:18 dist-test-slave-j0m4 krb5kdc[4071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
I20260923 17:22:18.289059 4161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379: Bootstrap starting.
I20260923 17:22:18.292555 4161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:18.293668 4161 log.cc:824] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:18.296643 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:18.274617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53316 (local address 127.28.72.254:38211)
0923 17:22:18.275512 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:18.275524 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:18.275563 (+ 39us) server_negotiation.cc:407] Connection header received
0923 17:22:18.277187 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:18.277211 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:18.277617 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:18.277983 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:18.279908 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:18.281304 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:18.283091 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:18.283871 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:18.284025 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:18.288887 (+ 4862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:18.288897 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:18.288915 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:18.288959 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:18.292259 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:18.293107 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:18.293115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:18.293122 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:18.293297 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:18.293745 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:18.293751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:18.293754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:18.294405 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:18.294756 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:18.295220 (+ 464us) server_negotiation.cc:299] Negotiation successful
0923 17:22:18.295648 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":470,"threads_started":1}
I20260923 17:22:18.296924 4161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379: No bootstrap required, opened a new log
I20260923 17:22:18.304880 4161 raft_consensus.cc:348] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e584ffce34a34c61b58a05e30d566379" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 38211 } }
I20260923 17:22:18.305338 4161 raft_consensus.cc:374] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:18.305457 4161 raft_consensus.cc:729] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e584ffce34a34c61b58a05e30d566379, State: Initialized, Role: FOLLOWER
I20260923 17:22:18.306015 4161 consensus_queue.cc:260] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e584ffce34a34c61b58a05e30d566379" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 38211 } }
I20260923 17:22:18.306277 4161 raft_consensus.cc:388] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:18.306591 4161 raft_consensus.cc:482] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:18.306783 4161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:18.309087 4161 raft_consensus.cc:504] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e584ffce34a34c61b58a05e30d566379" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 38211 } }
I20260923 17:22:18.309556 4161 leader_election.cc:302] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e584ffce34a34c61b58a05e30d566379; no voters:
I20260923 17:22:18.310079 4161 leader_election.cc:288] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:22:18.310214 4166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:18.310941 4166 raft_consensus.cc:686] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [term 1 LEADER]: Becoming Leader. State: Replica: e584ffce34a34c61b58a05e30d566379, State: Running, Role: LEADER
I20260923 17:22:18.311559 4166 consensus_queue.cc:237] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e584ffce34a34c61b58a05e30d566379" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 38211 } }
I20260923 17:22:18.312271 4161 sys_catalog.cc:564] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:18.314656 4168 sys_catalog.cc:455] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e584ffce34a34c61b58a05e30d566379. Latest consensus state: current_term: 1 leader_uuid: "e584ffce34a34c61b58a05e30d566379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e584ffce34a34c61b58a05e30d566379" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 38211 } } }
I20260923 17:22:18.315024 4168 sys_catalog.cc:458] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:18.316205 4167 sys_catalog.cc:455] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e584ffce34a34c61b58a05e30d566379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e584ffce34a34c61b58a05e30d566379" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 38211 } } }
I20260923 17:22:18.316844 4167 sys_catalog.cc:458] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:18.320186 4172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:18.325848 4172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:18.334906 4172 catalog_manager.cc:1347] Generated new cluster ID: 975aceca28c4443ba72954268d68d4fd
I20260923 17:22:18.335057 4172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:18.361970 4172 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:22:18.363636 4172 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:18.370620 4172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379: Generated new TSK 0
I20260923 17:22:18.371589 4172 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.193.keytab.
Entry for principal oryx/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.193.keytab.
I20260923 17:22:18.431627 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:38211
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.193.keytab
--principal=oryx/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:18.770632 4187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:18.770994 4187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:18.771235 4187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:18.778784 4187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:18.779187 4187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:18.779376 4187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:22:18.779528 4187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:22:18.789882 4187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.193.keytab
--principal=oryx/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.72.254:38211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:18.791409 4187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:18.792909 4187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:18.807039 4193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:18.807148 4195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:18.807102 4192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:18.808069 4187 server_base.cc:1034] running on GCE node
I20260923 17:22:18.808743 4187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:18.809733 4187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:18.810988 4187 hybrid_clock.cc:648] HybridClock initialized: now 1790184138810907 us; error 53 us; skew 500 ppm
I20260923 17:22:18.811419 4187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:18 dist-test-slave-j0m4 krb5kdc[4071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184138, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:18.815094 4187 init.cc:377] Logged in from keytab as oryx/127.28.72.193@KRBTEST.COM (short username oryx)
I20260923 17:22:18.816915 4187 webserver.cc:460] Webserver started at http://127.28.72.193:40593/ using document root <none> and password file <none>
I20260923 17:22:18.817665 4187 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:18.817752 4187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:18.818095 4187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:18.821139 4187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "3da3007573bc467fad9e2b65e6d76d36"
format_stamp: "Formatted at 2026-09-23 17:22:18 on dist-test-slave-j0m4"
server_key: "e656ade36953ed6f81af2c9457fd210c"
server_key_iv: "05f022407be4da5adeaa70ca55f03d75"
server_key_version: "encryptionkey@0"
I20260923 17:22:18.821971 4187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "3da3007573bc467fad9e2b65e6d76d36"
format_stamp: "Formatted at 2026-09-23 17:22:18 on dist-test-slave-j0m4"
server_key: "e656ade36953ed6f81af2c9457fd210c"
server_key_iv: "05f022407be4da5adeaa70ca55f03d75"
server_key_version: "encryptionkey@0"
I20260923 17:22:18.827801 4187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:22:18.831660 4202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:18.833253 4187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:22:18.833455 4187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "3da3007573bc467fad9e2b65e6d76d36"
format_stamp: "Formatted at 2026-09-23 17:22:18 on dist-test-slave-j0m4"
server_key: "e656ade36953ed6f81af2c9457fd210c"
server_key_iv: "05f022407be4da5adeaa70ca55f03d75"
server_key_version: "encryptionkey@0"
I20260923 17:22:18.833657 4187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:18.858248 4187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:18.862928 4187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:18.863200 4187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:18.866048 4187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:18.866160 4187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:18.866281 4187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:18.866357 4187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:22:18 dist-test-slave-j0m4 krb5kdc[4071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184138, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
I20260923 17:22:18.891009 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:18.866686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:56433 (local address 127.28.72.254:38211)
0923 17:22:18.867352 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:18.867357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:18.868530 (+ 1173us) server_negotiation.cc:407] Connection header received
0923 17:22:18.870652 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:18.870663 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:18.870756 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:18.870877 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:18.873222 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:18.874238 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:18.876994 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:18.877385 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:18.877536 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:18.883229 (+ 5693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:18.883235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:18.883238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:18.883272 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:18.887278 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:18.888381 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:18.888387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:18.888391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:18.888544 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:18.889193 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:18.889199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:18.889202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:18.889488 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:18.889646 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:18.890414 (+ 768us) server_negotiation.cc:299] Negotiation successful
0923 17:22:18.890639 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":453,"threads_started":1}
I20260923 17:22:18.892096 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:18.867488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:38211 (local address 127.28.72.193:56433)
0923 17:22:18.868346 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:18.868400 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:18.870371 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:18.871385 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:18.871398 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:18.872230 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:18.873034 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:18.873053 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:18.874413 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:18.874425 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:18.876757 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:18.876779 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:18.877646 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:18.877654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:18.878015 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:18.878909 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:18.878955 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:18.882836 (+ 3881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:18.887528 (+ 4692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:18.887546 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:18.887571 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:18.888239 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:18.888729 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:18.888737 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:18.888743 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:18.889055 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:18.889622 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:18.889631 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:18.890024 (+ 393us) client_negotiation.cc:769] Sending connection context
0923 17:22:18.890298 (+ 274us) client_negotiation.cc:240] Negotiation successful
0923 17:22:18.890854 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":353,"threads_started":1}
I20260923 17:22:18.913391 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:18.898563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:57881 (local address 127.28.72.254:38211)
0923 17:22:18.898763 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:18.898772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:18.899639 (+ 867us) server_negotiation.cc:407] Connection header received
0923 17:22:18.899828 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:18.899834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:18.899924 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:18.900029 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:18.900978 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:18.901679 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:18.902976 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:18.903478 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:18.903634 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:18.905998 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:18.906011 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:18.906015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:18.906047 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:18.909640 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:18.910703 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:18.910711 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:18.910717 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:18.910897 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:18.911529 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:18.911535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:18.911541 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:18.911969 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:18.912150 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:18.912689 (+ 539us) server_negotiation.cc:299] Negotiation successful
0923 17:22:18.912944 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 17:22:18.913564 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:18.898643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:38211 (local address 127.28.72.193:57881)
0923 17:22:18.899519 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:18.899535 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:18.899703 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:18.900055 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:18.900060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:18.900544 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:18.900834 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:18.900842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:18.901806 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:18.901814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:18.902818 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:18.902830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:18.903723 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:18.903733 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:18.904053 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:18.904871 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:18.904905 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:18.905792 (+ 887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:18.909801 (+ 4009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:18.909809 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:18.909816 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:18.910545 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:18.911045 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:18.911051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:18.911059 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:18.911346 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:18.912231 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:18.912239 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:18.912420 (+ 181us) client_negotiation.cc:769] Sending connection context
0923 17:22:18.912591 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 17:22:18.912969 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":614,"threads_started":1}
I20260923 17:22:18.923343 4187 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:37369
I20260923 17:22:18.923362 4322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:37369 every 8 connection(s)
I20260923 17:22:18.924618 4187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:22:18.926892 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 4187
I20260923 17:22:18.927006 4323 heartbeater.cc:347] Connected to a master server at 127.28.72.254:38211
I20260923 17:22:18.927161 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:22:18.927281 4323 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:18.927829 28963 external_mini_cluster.cc:1428] Setting key cc7c87c94379c745ab8506be7dd70b26
I20260923 17:22:18.927999 4323 heartbeater.cc:511] Master 127.28.72.254:38211 requested a full tablet report, sending...
I20260923 17:22:18.930137 4115 ts_manager.cc:194] Registered new tserver with Master: 3da3007573bc467fad9e2b65e6d76d36 (127.28.72.193:37369)
I20260923 17:22:18.931677 4115 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:56433
WARNING: no policy specified for oryx/127.28.72.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.72.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.194.keytab.
Entry for principal oryx/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.194.keytab.
I20260923 17:22:18.979936 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:38211
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.194.keytab
--principal=oryx/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:19.318585 4327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:19.318919 4327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:19.319099 4327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:19.326841 4327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:19.326993 4327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:19.327085 4327 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:22:19.327226 4327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:22:19.337452 4327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.194.keytab
--principal=oryx/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.72.254:38211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:19.338910 4327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:19.340371 4327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:19.353906 4332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:19.353914 4335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:19.353906 4333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:19.355233 4327 server_base.cc:1034] running on GCE node
I20260923 17:22:19.356009 4327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:19.357070 4327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:19.358340 4327 hybrid_clock.cc:648] HybridClock initialized: now 1790184139358270 us; error 42 us; skew 500 ppm
I20260923 17:22:19.358700 4327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:19 dist-test-slave-j0m4 krb5kdc[4071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184139, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:19.362182 4327 init.cc:377] Logged in from keytab as oryx/127.28.72.194@KRBTEST.COM (short username oryx)
I20260923 17:22:19.363751 4327 webserver.cc:460] Webserver started at http://127.28.72.194:35109/ using document root <none> and password file <none>
I20260923 17:22:19.364451 4327 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:19.364552 4327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:19.364830 4327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:19.367791 4327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "00b546d49dac43838cff9bc9a6cfe056"
format_stamp: "Formatted at 2026-09-23 17:22:19 on dist-test-slave-j0m4"
server_key: "64c2416a60d39a6eef2657ea09b7ebf0"
server_key_iv: "8fd730eecb608c5a6469b1c7f01652f4"
server_key_version: "encryptionkey@0"
I20260923 17:22:19.368539 4327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "00b546d49dac43838cff9bc9a6cfe056"
format_stamp: "Formatted at 2026-09-23 17:22:19 on dist-test-slave-j0m4"
server_key: "64c2416a60d39a6eef2657ea09b7ebf0"
server_key_iv: "8fd730eecb608c5a6469b1c7f01652f4"
server_key_version: "encryptionkey@0"
I20260923 17:22:19.373857 4327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:22:19.377581 4342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:19.379024 4327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:22:19.379262 4327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "00b546d49dac43838cff9bc9a6cfe056"
format_stamp: "Formatted at 2026-09-23 17:22:19 on dist-test-slave-j0m4"
server_key: "64c2416a60d39a6eef2657ea09b7ebf0"
server_key_iv: "8fd730eecb608c5a6469b1c7f01652f4"
server_key_version: "encryptionkey@0"
I20260923 17:22:19.379529 4327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:19.394593 4327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:19.399187 4327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:19.399492 4327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:19.402194 4327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:19.402274 4327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:19.402421 4327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:19.402521 4327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:22:19 dist-test-slave-j0m4 krb5kdc[4071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184139, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
I20260923 17:22:19.429812 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:19.403144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:43599 (local address 127.28.72.254:38211)
0923 17:22:19.403293 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:19.403297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:19.404944 (+ 1647us) server_negotiation.cc:407] Connection header received
0923 17:22:19.407395 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:19.407401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:19.407494 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:19.407610 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:19.410960 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.412028 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.414913 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.415714 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.415839 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:19.421689 (+ 5850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:19.421699 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:19.421707 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:19.421759 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:19.425224 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:19.426375 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:19.426384 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:19.426390 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:19.426577 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:19.427285 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:19.427294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:19.427300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:19.427749 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:19.427919 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:19.429044 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0923 17:22:19.429325 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 17:22:19.431897 4351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:19.403836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:38211 (local address 127.28.72.194:43599)
0923 17:22:19.404769 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:19.404808 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:19.407002 (+ 2194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:19.407946 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:19.407965 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:19.409188 (+ 1223us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:19.410689 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:19.410724 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.412233 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.412237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:19.414734 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:19.414747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.415968 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.415977 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:19.416383 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:19.417468 (+ 1085us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:19.417504 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:19.421390 (+ 3886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:19.425388 (+ 3998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:19.425399 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:19.425425 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:19.426216 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:19.426735 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:19.426745 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:19.426750 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:19.427096 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:19.427899 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:19.427905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:19.428504 (+ 599us) client_negotiation.cc:769] Sending connection context
0923 17:22:19.428878 (+ 374us) client_negotiation.cc:240] Negotiation successful
0923 17:22:19.429596 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":480,"threads_started":1}
I20260923 17:22:19.445521 4327 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:36227
I20260923 17:22:19.445562 4461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:36227 every 8 connection(s)
I20260923 17:22:19.446775 4327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:22:19.449105 4462 heartbeater.cc:347] Connected to a master server at 127.28.72.254:38211
I20260923 17:22:19.449817 4462 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:19.450670 4462 heartbeater.cc:511] Master 127.28.72.254:38211 requested a full tablet report, sending...
I20260923 17:22:19.452390 4115 ts_manager.cc:194] Registered new tserver with Master: 00b546d49dac43838cff9bc9a6cfe056 (127.28.72.194:36227)
I20260923 17:22:19.452659 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 4327
I20260923 17:22:19.452850 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:22:19.453320 28963 external_mini_cluster.cc:1428] Setting key 4ee86b404af9b044c50c7dc0239dc1da
I20260923 17:22:19.453490 4115 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:43599
I20260923 17:22:19.455260 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:19.439541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:57277 (local address 127.28.72.254:38211)
0923 17:22:19.439731 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:19.439735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:19.441271 (+ 1536us) server_negotiation.cc:407] Connection header received
0923 17:22:19.441496 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:19.441505 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:19.441598 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:19.441700 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:19.443487 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.444391 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.446290 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.446707 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.446844 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:19.448993 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:19.448998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:19.449001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:19.449060 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:19.452280 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:19.453129 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:19.453135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:19.453139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:19.453287 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:19.453717 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:19.453721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:19.453724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:19.454106 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:19.454263 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:19.454666 (+ 403us) server_negotiation.cc:299] Negotiation successful
0923 17:22:19.454872 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:22:19.455374 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:19.440147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:38211 (local address 127.28.72.194:57277)
0923 17:22:19.441174 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:19.441195 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:19.441346 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:19.441946 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:19.441953 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:19.442688 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:19.443259 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:19.443280 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.444556 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.444560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:19.446139 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:19.446151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.446912 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.446919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:19.447084 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:19.447905 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:19.447929 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:19.448793 (+ 864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:19.452433 (+ 3640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:19.452439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:19.452443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:19.453012 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:19.453415 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:19.453419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:19.453424 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:19.453579 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:19.454229 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:19.454234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:19.454425 (+ 191us) client_negotiation.cc:769] Sending connection context
0923 17:22:19.454574 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 17:22:19.454903 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"thread_start_us":759,"threads_started":1}
WARNING: no policy specified for oryx/127.28.72.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.72.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.195.keytab.
Entry for principal oryx/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.195.keytab.
I20260923 17:22:19.506551 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:38211
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.195.keytab
--principal=oryx/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:19.852993 4466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:19.853299 4466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:19.853475 4466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:19.861232 4466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:19.861375 4466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:19.861555 4466 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:22:19.861677 4466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:22:19.872253 4466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.195.keytab
--principal=oryx/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.72.254:38211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:19.873658 4466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:19.875064 4466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:19.889293 4472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:19.889245 4471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:19.889340 4474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:19.893214 4466 server_base.cc:1034] running on GCE node
I20260923 17:22:19.894081 4466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:19.895421 4466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:19.896679 4466 hybrid_clock.cc:648] HybridClock initialized: now 1790184139896569 us; error 65 us; skew 500 ppm
I20260923 17:22:19.897120 4466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:19 dist-test-slave-j0m4 krb5kdc[4071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184139, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:19.907322 4466 init.cc:377] Logged in from keytab as oryx/127.28.72.195@KRBTEST.COM (short username oryx)
I20260923 17:22:19.909040 4466 webserver.cc:460] Webserver started at http://127.28.72.195:32949/ using document root <none> and password file <none>
I20260923 17:22:19.909896 4466 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:19.910036 4466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:19.910368 4466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:19.913625 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "156308c205b841918853f294e89bd438"
format_stamp: "Formatted at 2026-09-23 17:22:19 on dist-test-slave-j0m4"
server_key: "5ad81837a31745ea8fbcbbc5855a2047"
server_key_iv: "592dfc731078fbd86abb2d0df3a47fd2"
server_key_version: "encryptionkey@0"
I20260923 17:22:19.914458 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "156308c205b841918853f294e89bd438"
format_stamp: "Formatted at 2026-09-23 17:22:19 on dist-test-slave-j0m4"
server_key: "5ad81837a31745ea8fbcbbc5855a2047"
server_key_iv: "592dfc731078fbd86abb2d0df3a47fd2"
server_key_version: "encryptionkey@0"
I20260923 17:22:19.920258 4466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:22:19.924401 4481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:19.925902 4466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:22:19.926095 4466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "156308c205b841918853f294e89bd438"
format_stamp: "Formatted at 2026-09-23 17:22:19 on dist-test-slave-j0m4"
server_key: "5ad81837a31745ea8fbcbbc5855a2047"
server_key_iv: "592dfc731078fbd86abb2d0df3a47fd2"
server_key_version: "encryptionkey@0"
I20260923 17:22:19.926328 4466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:19.934646 4323 heartbeater.cc:503] Master 127.28.72.254:38211 was elected leader, sending a full tablet report...
I20260923 17:22:19.946121 4466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:19.950623 4466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:19.950908 4466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:19.953806 4466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:19.953926 4466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:19.954078 4466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:19.954159 4466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:22:19 dist-test-slave-j0m4 krb5kdc[4071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184139, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
I20260923 17:22:19.982753 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:19.954752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:43767 (local address 127.28.72.254:38211)
0923 17:22:19.954946 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:19.954953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:19.956532 (+ 1579us) server_negotiation.cc:407] Connection header received
0923 17:22:19.958926 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:19.958936 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:19.959041 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:19.959115 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:19.962002 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.963247 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.967170 (+ 3923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.967790 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.967990 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:19.974648 (+ 6658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:19.974657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:19.974662 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:19.974716 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:19.977977 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:19.979386 (+ 1409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:19.979393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:19.979396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:19.979582 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:19.980390 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:19.980398 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:19.980404 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:19.980879 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:19.981085 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:19.981953 (+ 868us) server_negotiation.cc:299] Negotiation successful
0923 17:22:19.982280 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:22:19.984269 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:19.955448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:38211 (local address 127.28.72.195:43767)
0923 17:22:19.956341 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:19.956386 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:19.958516 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:19.959679 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:19.959697 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:19.960676 (+ 979us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:19.961748 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:19.961783 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.963524 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.963535 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:19.966870 (+ 3335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:19.966905 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.968164 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.968177 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:19.968642 (+ 465us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:19.969759 (+ 1117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:19.969805 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:19.974230 (+ 4425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:19.978209 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:19.978225 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:19.978256 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:19.979164 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:19.979757 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:19.979764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:19.979774 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:19.980206 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:19.981040 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:19.981047 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:19.981479 (+ 432us) client_negotiation.cc:769] Sending connection context
0923 17:22:19.981734 (+ 255us) client_negotiation.cc:240] Negotiation successful
0923 17:22:19.982362 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":369,"threads_started":1}
I20260923 17:22:20.005721 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:19.990719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:45121 (local address 127.28.72.254:38211)
0923 17:22:19.990904 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:19.990909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:19.991620 (+ 711us) server_negotiation.cc:407] Connection header received
0923 17:22:19.991814 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:19.991819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:19.991891 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:19.991997 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:19.992930 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.993629 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.995790 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.996428 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.996581 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:19.998591 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:19.998596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:19.998600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:19.998630 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:20.002101 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:20.003158 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:20.003165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:20.003169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:20.003344 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:20.003895 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:20.003904 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:20.003910 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:20.004412 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:20.004645 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:20.005078 (+ 433us) server_negotiation.cc:299] Negotiation successful
0923 17:22:20.005340 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:22:20.005990 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:19.990831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:38211 (local address 127.28.72.195:45121)
0923 17:22:19.991510 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:19.991528 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:19.991692 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:19.992019 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:19.992024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:19.992503 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:19.992784 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:19.992792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.993758 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.993766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:19.995651 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:19.995669 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:19.996628 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:19.996635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:19.996814 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:19.997574 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:19.997591 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:19.998376 (+ 785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:20.002274 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:20.002282 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:20.002287 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:20.002965 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:20.003471 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:20.003477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:20.003480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:20.003724 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:20.004569 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:20.004573 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:20.004748 (+ 175us) client_negotiation.cc:769] Sending connection context
0923 17:22:20.004977 (+ 229us) client_negotiation.cc:240] Negotiation successful
0923 17:22:20.005411 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":405,"threads_started":1}
I20260923 17:22:20.014068 4466 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:40547
I20260923 17:22:20.014091 4600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:40547 every 8 connection(s)
I20260923 17:22:20.015517 4466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:22:20.017910 4601 heartbeater.cc:347] Connected to a master server at 127.28.72.254:38211
I20260923 17:22:20.018185 4601 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:20.018918 4601 heartbeater.cc:511] Master 127.28.72.254:38211 requested a full tablet report, sending...
I20260923 17:22:20.020319 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 4466
I20260923 17:22:20.020505 4115 ts_manager.cc:194] Registered new tserver with Master: 156308c205b841918853f294e89bd438 (127.28.72.195:40547)
I20260923 17:22:20.020599 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:22:20.021086 28963 external_mini_cluster.cc:1428] Setting key 70f2321d893d6fc0a59691efaf700a6d
I20260923 17:22:20.021677 4115 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:43767
I20260923 17:22:20.025074 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:22:20 dist-test-slave-j0m4 krb5kdc[4071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184140, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:22:20 dist-test-slave-j0m4 krb5kdc[4071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:20 dist-test-slave-j0m4 krb5kdc[4071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:20.457698 4462 heartbeater.cc:503] Master 127.28.72.254:38211 was elected leader, sending a full tablet report...
I20260923 17:22:21.025285 4601 heartbeater.cc:503] Master 127.28.72.254:38211 was elected leader, sending a full tablet report...
W20260923 17:22:21.157636 4609 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.074s user 0.005s sys 0.009s
W20260923 17:22:21.158880 4211 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:20.078343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.28.72.254:38211)
0923 17:22:20.078571 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:20.078579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:20.078767 (+ 188us) server_negotiation.cc:407] Connection header received
0923 17:22:20.078984 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:20.078991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:20.079060 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:20.079200 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:20.080573 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:20.081304 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:20.082548 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:20.083031 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:20.083229 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.158541 (+1075312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53318: BlockingRecv error: recv got EOF from 127.0.0.1:53318 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 23 17:22:21 dist-test-slave-j0m4 krb5kdc[4071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184140, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
I20260923 17:22:21.188911 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.165254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.28.72.254:38211)
0923 17:22:21.165592 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:21.165601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:21.165897 (+ 296us) server_negotiation.cc:407] Connection header received
0923 17:22:21.166165 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:21.166170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:21.166276 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:21.166402 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:21.167871 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.168676 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.170359 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.170849 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.170998 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.183492 (+ 12494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:21.183500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:21.183504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:21.183546 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:21.186270 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:21.187235 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:21.187240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:21.187242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:21.187370 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:21.187759 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:21.187762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:21.187764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:21.188038 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:21.188260 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:21.188514 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 17:22:21.188683 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260923 17:22:21.193756 4115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:22:21.201139 4115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:22:21.224985 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.213889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:37369 (local address 127.0.0.1:53688)
0923 17:22:21.215205 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:21.215246 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:21.215504 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:21.216570 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:21.216580 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:21.216857 (+ 277us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:21.217490 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.217506 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.220105 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.220110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.222204 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.222220 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.223948 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.223954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.224131 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.224189 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 17:22:21.224342 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 17:22:21.224532 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"spinlock_wait_cycles":472448,"thread_start_us":1528,"threads_started":1}
I20260923 17:22:21.226176 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.214582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.28.72.195:40547)
0923 17:22:21.215244 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:21.215255 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:21.215875 (+ 620us) server_negotiation.cc:407] Connection header received
0923 17:22:21.217731 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:21.217736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:21.217901 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:21.218043 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:21.218765 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.220283 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.222960 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.224469 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.224618 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.224820 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:21.225687 (+ 867us) server_negotiation.cc:299] Negotiation successful
0923 17:22:21.225870 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":1413,"threads_started":1}
I20260923 17:22:21.226295 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.215647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:36227 (local address 127.0.0.1:38216)
0923 17:22:21.217445 (+ 1798us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:21.217466 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:21.217576 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:21.218181 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:21.218185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:21.218212 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:21.218774 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.218787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.220706 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.220713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.223048 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.223062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.225149 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.225154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.225301 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.225420 (+ 119us) client_negotiation.cc:769] Sending connection context
0923 17:22:21.225557 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 17:22:21.225700 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1644,"mutex_wait_us":71,"thread_start_us":1510,"threads_started":1}
I20260923 17:22:21.226425 4620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.214021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:40547 (local address 127.0.0.1:46196)
0923 17:22:21.215712 (+ 1691us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:21.215738 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:21.216823 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:21.218080 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:21.218088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:21.218116 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:21.218605 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.218619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.220411 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.220419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.222768 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.222781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.224987 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.224993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.225426 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.225453 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:22:21.225543 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 17:22:21.225757 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1570,"mutex_wait_us":113,"thread_start_us":781,"threads_started":1}
I20260923 17:22:21.226846 4621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.214102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.28.72.193:37369)
0923 17:22:21.214645 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:21.214654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:21.215340 (+ 686us) server_negotiation.cc:407] Connection header received
0923 17:22:21.215755 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:21.215766 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:21.216040 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:21.216217 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:21.218292 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.219922 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.222365 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.223739 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.223897 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.224131 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:21.224386 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 17:22:21.224564 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":231,"threads_started":1}
I20260923 17:22:21.227340 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.216202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.28.72.194:36227)
0923 17:22:21.217707 (+ 1505us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:21.217719 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:21.217740 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:22:21.217813 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:21.217818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:21.217966 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:21.218175 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:21.218991 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.220575 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.223276 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.224701 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.224825 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.225026 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:21.226742 (+ 1716us) server_negotiation.cc:299] Negotiation successful
0923 17:22:21.226947 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1354,"thread_start_us":1221,"threads_started":1}
I20260923 17:22:21.231144 4253 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfa8a36cc714bc991dc0c0ce769092e (DEFAULT_TABLE table=test-table [id=f3e1999135f2417b8eedcf07798845e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:21.232774 4253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfa8a36cc714bc991dc0c0ce769092e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:21.231148 4392 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfa8a36cc714bc991dc0c0ce769092e (DEFAULT_TABLE table=test-table [id=f3e1999135f2417b8eedcf07798845e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:21.234066 4392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfa8a36cc714bc991dc0c0ce769092e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:21.234958 4531 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfa8a36cc714bc991dc0c0ce769092e (DEFAULT_TABLE table=test-table [id=f3e1999135f2417b8eedcf07798845e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:21.236562 4531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfa8a36cc714bc991dc0c0ce769092e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:21.250011 4627 tablet_bootstrap.cc:492] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056: Bootstrap starting.
I20260923 17:22:21.252610 4626 tablet_bootstrap.cc:492] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36: Bootstrap starting.
I20260923 17:22:21.253582 4628 tablet_bootstrap.cc:492] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438: Bootstrap starting.
I20260923 17:22:21.254197 4627 tablet_bootstrap.cc:654] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:21.255856 4626 tablet_bootstrap.cc:654] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:21.255854 4627 log.cc:824] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:21.256824 4628 tablet_bootstrap.cc:654] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:21.257052 4626 log.cc:824] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:21.258003 4628 log.cc:824] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:21.259853 4627 tablet_bootstrap.cc:492] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056: No bootstrap required, opened a new log
I20260923 17:22:21.260142 4627 ts_tablet_manager.cc:1397] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 17:22:21.261022 4626 tablet_bootstrap.cc:492] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36: No bootstrap required, opened a new log
I20260923 17:22:21.261025 4628 tablet_bootstrap.cc:492] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438: No bootstrap required, opened a new log
I20260923 17:22:21.261278 4628 ts_tablet_manager.cc:1397] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260923 17:22:21.261278 4626 ts_tablet_manager.cc:1397] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260923 17:22:21.268014 4627 raft_consensus.cc:348] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } }
I20260923 17:22:21.268515 4627 raft_consensus.cc:374] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:21.268602 4627 raft_consensus.cc:729] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b546d49dac43838cff9bc9a6cfe056, State: Initialized, Role: FOLLOWER
I20260923 17:22:21.269302 4627 consensus_queue.cc:260] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } }
I20260923 17:22:21.272198 4628 raft_consensus.cc:348] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } }
I20260923 17:22:21.272198 4626 raft_consensus.cc:348] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } }
I20260923 17:22:21.272658 4626 raft_consensus.cc:374] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:21.272658 4628 raft_consensus.cc:374] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:21.272760 4626 raft_consensus.cc:729] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da3007573bc467fad9e2b65e6d76d36, State: Initialized, Role: FOLLOWER
I20260923 17:22:21.272753 4628 raft_consensus.cc:729] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156308c205b841918853f294e89bd438, State: Initialized, Role: FOLLOWER
I20260923 17:22:21.273504 4628 consensus_queue.cc:260] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } }
I20260923 17:22:21.273504 4626 consensus_queue.cc:260] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } }
I20260923 17:22:21.279095 4628 ts_tablet_manager.cc:1428] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
I20260923 17:22:21.289553 4626 ts_tablet_manager.cc:1428] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36: Time spent starting tablet: real 0.028s user 0.017s sys 0.011s
I20260923 17:22:21.291160 4627 ts_tablet_manager.cc:1428] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056: Time spent starting tablet: real 0.031s user 0.023s sys 0.008s
I20260923 17:22:21.313586 4632 raft_consensus.cc:482] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:21.313936 4632 raft_consensus.cc:504] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } }
I20260923 17:22:21.315970 4632 leader_election.cc:288] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da3007573bc467fad9e2b65e6d76d36 (127.28.72.193:37369), 156308c205b841918853f294e89bd438 (127.28.72.195:40547)
I20260923 17:22:21.324940 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.316600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:55057 (local address 127.28.72.195:40547)
0923 17:22:21.316835 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:21.316842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:21.317357 (+ 515us) server_negotiation.cc:407] Connection header received
0923 17:22:21.317545 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:21.317549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:21.317614 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:21.317711 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:21.318305 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.319006 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.322091 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.323563 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.323763 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.323849 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:21.324503 (+ 654us) server_negotiation.cc:299] Negotiation successful
0923 17:22:21.324661 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:22:21.324940 4621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.316264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:41585 (local address 127.28.72.193:37369)
0923 17:22:21.316478 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:21.316487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:21.317511 (+ 1024us) server_negotiation.cc:407] Connection header received
0923 17:22:21.317619 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:21.317624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:21.317701 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:21.317794 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:21.318374 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.319079 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.322085 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.323563 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.323769 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.323849 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:21.324503 (+ 654us) server_negotiation.cc:299] Negotiation successful
0923 17:22:21.324661 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:22:21.325162 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.316238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:37369 (local address 127.28.72.194:41585)
0923 17:22:21.317357 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:21.317382 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:21.317509 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:21.317821 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:21.317826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:21.317865 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:21.318221 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.318230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.319248 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.319256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.321875 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.321892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.323948 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.323956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.324223 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.324299 (+ 76us) client_negotiation.cc:769] Sending connection context
0923 17:22:21.324410 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 17:22:21.324634 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":289,"thread_start_us":691,"threads_started":1}
I20260923 17:22:21.325591 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.316207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:40547 (local address 127.28.72.194:55057)
0923 17:22:21.317254 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:21.317269 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:21.317389 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:21.317726 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:21.317730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:21.317760 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:21.318151 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.318161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.319169 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.319176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.321880 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.321892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.323959 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.323964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.324217 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.324297 (+ 80us) client_negotiation.cc:769] Sending connection context
0923 17:22:21.324410 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 17:22:21.324634 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":162,"thread_start_us":727,"threads_started":1}
I20260923 17:22:21.326329 4278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfa8a36cc714bc991dc0c0ce769092e" candidate_uuid: "00b546d49dac43838cff9bc9a6cfe056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da3007573bc467fad9e2b65e6d76d36" is_pre_election: true
I20260923 17:22:21.326802 4278 raft_consensus.cc:2454] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b546d49dac43838cff9bc9a6cfe056 in term 0.
I20260923 17:22:21.327492 4345 leader_election.cc:302] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b546d49dac43838cff9bc9a6cfe056, 3da3007573bc467fad9e2b65e6d76d36; no voters:
I20260923 17:22:21.327929 4632 raft_consensus.cc:2781] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:22:21.327842 4551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfa8a36cc714bc991dc0c0ce769092e" candidate_uuid: "00b546d49dac43838cff9bc9a6cfe056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156308c205b841918853f294e89bd438" is_pre_election: true
I20260923 17:22:21.328101 4632 raft_consensus.cc:482] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:22:21.328212 4632 raft_consensus.cc:3037] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:21.328363 4551 raft_consensus.cc:2454] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b546d49dac43838cff9bc9a6cfe056 in term 0.
I20260923 17:22:21.330930 4632 raft_consensus.cc:504] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } }
I20260923 17:22:21.331739 4632 leader_election.cc:288] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [CANDIDATE]: Term 1 election: Requested vote from peers 3da3007573bc467fad9e2b65e6d76d36 (127.28.72.193:37369), 156308c205b841918853f294e89bd438 (127.28.72.195:40547)
I20260923 17:22:21.332254 4278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfa8a36cc714bc991dc0c0ce769092e" candidate_uuid: "00b546d49dac43838cff9bc9a6cfe056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da3007573bc467fad9e2b65e6d76d36"
I20260923 17:22:21.332337 4551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfa8a36cc714bc991dc0c0ce769092e" candidate_uuid: "00b546d49dac43838cff9bc9a6cfe056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156308c205b841918853f294e89bd438"
I20260923 17:22:21.332490 4278 raft_consensus.cc:3037] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:21.332640 4551 raft_consensus.cc:3037] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:21.334947 4551 raft_consensus.cc:2454] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b546d49dac43838cff9bc9a6cfe056 in term 1.
I20260923 17:22:21.335359 4278 raft_consensus.cc:2454] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b546d49dac43838cff9bc9a6cfe056 in term 1.
I20260923 17:22:21.335530 4346 leader_election.cc:302] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b546d49dac43838cff9bc9a6cfe056, 156308c205b841918853f294e89bd438; no voters:
I20260923 17:22:21.335947 4632 raft_consensus.cc:2781] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:21.336443 4632 raft_consensus.cc:686] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [term 1 LEADER]: Becoming Leader. State: Replica: 00b546d49dac43838cff9bc9a6cfe056, State: Running, Role: LEADER
I20260923 17:22:21.337074 4632 consensus_queue.cc:237] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } }
I20260923 17:22:21.342454 4116 catalog_manager.cc:5495] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b546d49dac43838cff9bc9a6cfe056 (127.28.72.194). New cstate: current_term: 1 leader_uuid: "00b546d49dac43838cff9bc9a6cfe056" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } health_report { overall_health: HEALTHY } } }
I20260923 17:22:21.382871 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.377186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.28.72.194:36227)
0923 17:22:21.377395 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:21.377399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:21.377418 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:22:21.377680 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:21.377684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:21.377772 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:21.377901 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:21.378227 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.378994 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.380450 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.380851 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.380946 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.381492 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:21.382119 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:21.382243 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:21.382469 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 17:22:21.382541 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 17:22:21.392129 4278 raft_consensus.cc:1261] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Refusing update from remote peer 00b546d49dac43838cff9bc9a6cfe056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:21.392129 4551 raft_consensus.cc:1261] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Refusing update from remote peer 00b546d49dac43838cff9bc9a6cfe056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:21.393330 4637 consensus_queue.cc:1035] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:21.393774 4632 consensus_queue.cc:1035] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:21.407773 4640 mvcc.cc:205] Tried to move back new op lower bound from 7332594243130384384 to 7332594242921631744. Current Snapshot: MvccSnapshot[applied={T|T < 7332594243130384384}]
I20260923 17:22:21.423058 4115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:21.423429 4115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:21.427107 4115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379: Sending DeleteTablet for 3 replicas of tablet 9bfa8a36cc714bc991dc0c0ce769092e
I20260923 17:22:21.428650 4253 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfa8a36cc714bc991dc0c0ce769092e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:21 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:53688
I20260923 17:22:21.428925 4531 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfa8a36cc714bc991dc0c0ce769092e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:21 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:46196
I20260923 17:22:21.429524 4392 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfa8a36cc714bc991dc0c0ce769092e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:21 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:38216
I20260923 17:22:21.429715 4650 tablet_replica.cc:330] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36: stopping tablet replica
I20260923 17:22:21.430227 4650 raft_consensus.cc:2229] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:21.430774 4650 raft_consensus.cc:2258] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:21.431730 4115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:22:21.432384 4115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:22:21.432926 4650 ts_tablet_manager.cc:1905] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:21.434900 4652 tablet_replica.cc:330] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056: stopping tablet replica
I20260923 17:22:21.438102 4652 raft_consensus.cc:2229] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:22:21.438779 4651 tablet_replica.cc:330] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438: stopping tablet replica
I20260923 17:22:21.438818 4652 raft_consensus.cc:2258] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:21.439704 4651 raft_consensus.cc:2229] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:21.440223 4651 raft_consensus.cc:2258] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:21.441138 4652 ts_tablet_manager.cc:1905] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:21.442648 4392 tablet_service.cc:1467] Processing CreateTablet for tablet 194712753ce84ae8b0ce5161724ea074 (DEFAULT_TABLE table=test-table [id=e3bdcc17b70c469ea1ec76bf5ac910b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:21.442653 4253 tablet_service.cc:1467] Processing CreateTablet for tablet 194712753ce84ae8b0ce5161724ea074 (DEFAULT_TABLE table=test-table [id=e3bdcc17b70c469ea1ec76bf5ac910b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:21.443481 4253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194712753ce84ae8b0ce5161724ea074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:21.443611 4531 tablet_service.cc:1467] Processing CreateTablet for tablet 194712753ce84ae8b0ce5161724ea074 (DEFAULT_TABLE table=test-table [id=e3bdcc17b70c469ea1ec76bf5ac910b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:21.444283 4531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194712753ce84ae8b0ce5161724ea074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:21.444543 4392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194712753ce84ae8b0ce5161724ea074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:21.453640 4626 tablet_bootstrap.cc:492] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36: Bootstrap starting.
I20260923 17:22:21.457118 4626 tablet_bootstrap.cc:654] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:21.458633 4628 tablet_bootstrap.cc:492] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438: Bootstrap starting.
I20260923 17:22:21.459147 4627 tablet_bootstrap.cc:492] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056: Bootstrap starting.
I20260923 17:22:21.460974 4628 tablet_bootstrap.cc:654] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:21.461520 4627 tablet_bootstrap.cc:654] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:21.462074 4650 ts_tablet_manager.cc:1918] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:21.462263 4650 log.cc:1196] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/9bfa8a36cc714bc991dc0c0ce769092e
I20260923 17:22:21.462953 4650 ts_tablet_manager.cc:1939] T 9bfa8a36cc714bc991dc0c0ce769092e P 3da3007573bc467fad9e2b65e6d76d36: Deleting consensus metadata
I20260923 17:22:21.465452 4102 catalog_manager.cc:4841] TS 3da3007573bc467fad9e2b65e6d76d36 (127.28.72.193:37369): tablet 9bfa8a36cc714bc991dc0c0ce769092e (table test-table [id=f3e1999135f2417b8eedcf07798845e6]) successfully deleted
I20260923 17:22:21.472960 4627 tablet_bootstrap.cc:492] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056: No bootstrap required, opened a new log
I20260923 17:22:21.473311 4627 ts_tablet_manager.cc:1397] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.007s
I20260923 17:22:21.474189 4628 tablet_bootstrap.cc:492] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438: No bootstrap required, opened a new log
I20260923 17:22:21.474401 4628 ts_tablet_manager.cc:1397] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20260923 17:22:21.474713 4627 raft_consensus.cc:348] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.475068 4627 raft_consensus.cc:374] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:21.475281 4627 raft_consensus.cc:729] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b546d49dac43838cff9bc9a6cfe056, State: Initialized, Role: FOLLOWER
I20260923 17:22:21.475720 4627 consensus_queue.cc:260] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.476472 4627 ts_tablet_manager.cc:1428] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260923 17:22:21.477030 4628 raft_consensus.cc:348] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.477289 4628 raft_consensus.cc:374] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:21.477425 4628 raft_consensus.cc:729] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156308c205b841918853f294e89bd438, State: Initialized, Role: FOLLOWER
I20260923 17:22:21.477843 4652 ts_tablet_manager.cc:1918] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:21.477768 4628 consensus_queue.cc:260] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.479290 4651 ts_tablet_manager.cc:1905] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:21.479570 4652 log.cc:1196] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/9bfa8a36cc714bc991dc0c0ce769092e
I20260923 17:22:21.480360 4652 ts_tablet_manager.cc:1939] T 9bfa8a36cc714bc991dc0c0ce769092e P 00b546d49dac43838cff9bc9a6cfe056: Deleting consensus metadata
I20260923 17:22:21.481848 4626 tablet_bootstrap.cc:492] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36: No bootstrap required, opened a new log
I20260923 17:22:21.482029 4626 ts_tablet_manager.cc:1397] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.000s
I20260923 17:22:21.482163 4103 catalog_manager.cc:4841] TS 00b546d49dac43838cff9bc9a6cfe056 (127.28.72.194:36227): tablet 9bfa8a36cc714bc991dc0c0ce769092e (table test-table [id=f3e1999135f2417b8eedcf07798845e6]) successfully deleted
I20260923 17:22:21.481019 4628 ts_tablet_manager.cc:1428] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260923 17:22:21.484315 4626 raft_consensus.cc:348] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.484570 4626 raft_consensus.cc:374] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:21.484673 4626 raft_consensus.cc:729] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da3007573bc467fad9e2b65e6d76d36, State: Initialized, Role: FOLLOWER
I20260923 17:22:21.484997 4626 consensus_queue.cc:260] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.485713 4626 ts_tablet_manager.cc:1428] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 17:22:21.490496 4651 ts_tablet_manager.cc:1918] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:21.490660 4651 log.cc:1196] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/9bfa8a36cc714bc991dc0c0ce769092e
I20260923 17:22:21.491266 4651 ts_tablet_manager.cc:1939] T 9bfa8a36cc714bc991dc0c0ce769092e P 156308c205b841918853f294e89bd438: Deleting consensus metadata
I20260923 17:22:21.493005 4103 catalog_manager.cc:4841] TS 156308c205b841918853f294e89bd438 (127.28.72.195:40547): tablet 9bfa8a36cc714bc991dc0c0ce769092e (table test-table [id=f3e1999135f2417b8eedcf07798845e6]) successfully deleted
W20260923 17:22:21.518229 4602 tablet.cc:2369] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:22:21.519140 4634 raft_consensus.cc:482] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:21.519376 4634 raft_consensus.cc:504] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.521472 4634 leader_election.cc:288] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da3007573bc467fad9e2b65e6d76d36 (127.28.72.193:37369), 00b546d49dac43838cff9bc9a6cfe056 (127.28.72.194:36227)
I20260923 17:22:21.531734 4621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.521838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:33551 (local address 127.28.72.193:37369)
0923 17:22:21.522051 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:21.522057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:21.523691 (+ 1634us) server_negotiation.cc:407] Connection header received
0923 17:22:21.523873 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:21.523881 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:21.523986 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:21.524109 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:21.524750 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.526291 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.529115 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.530444 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.530694 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.530785 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:21.531245 (+ 460us) server_negotiation.cc:299] Negotiation successful
0923 17:22:21.531452 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:22:21.531740 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.521827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:37369 (local address 127.28.72.195:33551)
0923 17:22:21.523567 (+ 1740us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:21.523587 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:21.523730 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:21.524144 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:21.524149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:21.524179 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:21.524561 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.524573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.526489 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.526496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.528881 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.528903 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.530781 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.530788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.531025 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.531060 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:22:21.531209 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 17:22:21.531358 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1613,"thread_start_us":1442,"threads_started":1}
I20260923 17:22:21.532932 4278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194712753ce84ae8b0ce5161724ea074" candidate_uuid: "156308c205b841918853f294e89bd438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da3007573bc467fad9e2b65e6d76d36" is_pre_election: true
I20260923 17:22:21.533214 4278 raft_consensus.cc:2454] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 156308c205b841918853f294e89bd438 in term 0.
I20260923 17:22:21.533712 4484 leader_election.cc:302] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156308c205b841918853f294e89bd438, 3da3007573bc467fad9e2b65e6d76d36; no voters:
I20260923 17:22:21.534241 4634 raft_consensus.cc:2781] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:22:21.534403 4634 raft_consensus.cc:482] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:22:21.534526 4634 raft_consensus.cc:3037] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:21.537376 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.522518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:37613 (local address 127.28.72.194:36227)
0923 17:22:21.522682 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:21.522688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:21.529729 (+ 7041us) server_negotiation.cc:407] Connection header received
0923 17:22:21.529915 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:21.529920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:21.530108 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:21.530211 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:21.531031 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.532182 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.534763 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.536124 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.536301 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.536384 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:21.536816 (+ 432us) server_negotiation.cc:299] Negotiation successful
0923 17:22:21.537050 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:22:21.537935 4634 raft_consensus.cc:504] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.538364 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.522324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:36227 (local address 127.28.72.195:37613)
0923 17:22:21.529567 (+ 7243us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:21.529589 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:21.529777 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:21.530295 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:21.530300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:21.530333 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:21.530819 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.530833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.532323 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.532329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.534493 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.534508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.536376 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.536384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.536599 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.536643 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 17:22:21.537641 (+ 998us) client_negotiation.cc:240] Negotiation successful
0923 17:22:21.537802 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7101,"mutex_wait_us":1,"thread_start_us":6885,"threads_started":1}
I20260923 17:22:21.538717 4634 leader_election.cc:288] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [CANDIDATE]: Term 1 election: Requested vote from peers 3da3007573bc467fad9e2b65e6d76d36 (127.28.72.193:37369), 00b546d49dac43838cff9bc9a6cfe056 (127.28.72.194:36227)
I20260923 17:22:21.539435 4412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194712753ce84ae8b0ce5161724ea074" candidate_uuid: "156308c205b841918853f294e89bd438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b546d49dac43838cff9bc9a6cfe056" is_pre_election: true
I20260923 17:22:21.539583 4411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194712753ce84ae8b0ce5161724ea074" candidate_uuid: "156308c205b841918853f294e89bd438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b546d49dac43838cff9bc9a6cfe056"
I20260923 17:22:21.539953 4412 raft_consensus.cc:2454] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 156308c205b841918853f294e89bd438 in term 0.
I20260923 17:22:21.540526 4278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194712753ce84ae8b0ce5161724ea074" candidate_uuid: "156308c205b841918853f294e89bd438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da3007573bc467fad9e2b65e6d76d36"
I20260923 17:22:21.540762 4278 raft_consensus.cc:3037] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:21.542809 4278 raft_consensus.cc:2454] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 156308c205b841918853f294e89bd438 in term 1.
I20260923 17:22:21.543277 4484 leader_election.cc:302] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 156308c205b841918853f294e89bd438, 3da3007573bc467fad9e2b65e6d76d36; no voters: 00b546d49dac43838cff9bc9a6cfe056
I20260923 17:22:21.543628 4634 raft_consensus.cc:2781] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:21.544080 4634 raft_consensus.cc:686] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [term 1 LEADER]: Becoming Leader. State: Replica: 156308c205b841918853f294e89bd438, State: Running, Role: LEADER
I20260923 17:22:21.544515 4634 consensus_queue.cc:237] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.548491 4116 catalog_manager.cc:5495] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 reported cstate change: term changed from 0 to 1, leader changed from <none> to 156308c205b841918853f294e89bd438 (127.28.72.195). New cstate: current_term: 1 leader_uuid: "156308c205b841918853f294e89bd438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } health_report { overall_health: HEALTHY } } }
Sep 23 17:22:21 dist-test-slave-j0m4 krb5kdc[4071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184138, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.254@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
I20260923 17:22:21.609494 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.593805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.28.72.254:38211)
0923 17:22:21.594084 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:21.594092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:21.594959 (+ 867us) server_negotiation.cc:407] Connection header received
0923 17:22:21.595203 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:21.595208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:21.595271 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:21.595377 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:21.596619 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.597402 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.598690 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.599083 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.599299 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.603537 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:21.603541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:21.603544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:21.603573 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:21.606543 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:21.607396 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:21.607403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:21.607409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:21.607560 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:21.607932 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:21.607937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:21.607940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:21.608201 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:21.608368 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:21.608817 (+ 449us) server_negotiation.cc:299] Negotiation successful
0923 17:22:21.609135 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":133}
I20260923 17:22:21.609516 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.593990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:38211 (local address 127.0.0.1:53340)
0923 17:22:21.594815 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:21.594847 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:21.595039 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:21.595405 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:21.595410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:21.596214 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:21.596468 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.596478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.597535 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.597542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.598554 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.598567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.599316 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.599322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.599478 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.600240 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:21.600257 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:21.603311 (+ 3054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:21.606735 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:21.606750 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:21.606760 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:21.607274 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:21.607666 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:21.607673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:21.607677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:21.607816 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:21.608325 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:21.608339 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:21.608522 (+ 183us) client_negotiation.cc:769] Sending connection context
0923 17:22:21.608708 (+ 186us) client_negotiation.cc:240] Negotiation successful
0923 17:22:21.609015 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":548,"threads_started":1}
I20260923 17:22:21.614995 4116 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:53340:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:22:21.616137 4116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:22:21.623286 4253 tablet_service.cc:1467] Processing CreateTablet for tablet 6a17313e24ac4770af725fa26a3de1f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac294499fed74668aa853068292518e4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:22:21.623641 4392 tablet_service.cc:1467] Processing CreateTablet for tablet 6a17313e24ac4770af725fa26a3de1f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac294499fed74668aa853068292518e4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:22:21.623907 4253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a17313e24ac4770af725fa26a3de1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:21.624225 4392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a17313e24ac4770af725fa26a3de1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:21.627676 4531 tablet_service.cc:1467] Processing CreateTablet for tablet 6a17313e24ac4770af725fa26a3de1f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac294499fed74668aa853068292518e4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:22:21.628225 4531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a17313e24ac4770af725fa26a3de1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:21.631398 4626 tablet_bootstrap.cc:492] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36: Bootstrap starting.
I20260923 17:22:21.632792 4626 tablet_bootstrap.cc:654] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:21.633796 4627 tablet_bootstrap.cc:492] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056: Bootstrap starting.
I20260923 17:22:21.635403 4627 tablet_bootstrap.cc:654] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:21.635864 4628 tablet_bootstrap.cc:492] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438: Bootstrap starting.
I20260923 17:22:21.638006 4628 tablet_bootstrap.cc:654] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:21.638747 4626 tablet_bootstrap.cc:492] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36: No bootstrap required, opened a new log
I20260923 17:22:21.638988 4626 ts_tablet_manager.cc:1397] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260923 17:22:21.640070 4626 raft_consensus.cc:348] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.640414 4626 raft_consensus.cc:374] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:21.640516 4626 raft_consensus.cc:729] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da3007573bc467fad9e2b65e6d76d36, State: Initialized, Role: FOLLOWER
I20260923 17:22:21.641201 4626 consensus_queue.cc:260] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.641561 4627 tablet_bootstrap.cc:492] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056: No bootstrap required, opened a new log
I20260923 17:22:21.641773 4627 ts_tablet_manager.cc:1397] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260923 17:22:21.641812 4633 tablet_replica.cc:439] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } }
I20260923 17:22:21.642256 4633 tablet_replica.cc:442] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:22:21.644344 4627 raft_consensus.cc:348] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.644675 4627 raft_consensus.cc:374] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:21.644804 4627 raft_consensus.cc:729] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b546d49dac43838cff9bc9a6cfe056, State: Initialized, Role: FOLLOWER
I20260923 17:22:21.645103 4627 consensus_queue.cc:260] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.645623 4645 tablet_replica.cc:439] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } }
I20260923 17:22:21.645969 4626 ts_tablet_manager.cc:1428] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260923 17:22:21.646095 4645 tablet_replica.cc:442] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:22:21.648715 4627 ts_tablet_manager.cc:1428] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260923 17:22:21.651073 4628 tablet_bootstrap.cc:492] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438: No bootstrap required, opened a new log
I20260923 17:22:21.651327 4628 ts_tablet_manager.cc:1397] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.011s
I20260923 17:22:21.652447 4628 raft_consensus.cc:348] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.652649 4628 raft_consensus.cc:374] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:21.652743 4628 raft_consensus.cc:729] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156308c205b841918853f294e89bd438, State: Initialized, Role: FOLLOWER
I20260923 17:22:21.652983 4628 consensus_queue.cc:260] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.653391 4634 tablet_replica.cc:439] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } }
I20260923 17:22:21.653582 4628 ts_tablet_manager.cc:1428] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260923 17:22:21.653690 4634 tablet_replica.cc:442] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438: This TxnStatusTablet replica's current role is: FOLLOWER
W20260923 17:22:21.679248 4324 tablet.cc:2369] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:22:21.701277 4463 tablet.cc:2369] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:22:21.712392 4634 raft_consensus.cc:482] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:21.712801 4634 raft_consensus.cc:504] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.713562 4634 leader_election.cc:288] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da3007573bc467fad9e2b65e6d76d36 (127.28.72.193:37369), 00b546d49dac43838cff9bc9a6cfe056 (127.28.72.194:36227)
I20260923 17:22:21.714191 4278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a17313e24ac4770af725fa26a3de1f8" candidate_uuid: "156308c205b841918853f294e89bd438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da3007573bc467fad9e2b65e6d76d36" is_pre_election: true
I20260923 17:22:21.714223 4411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a17313e24ac4770af725fa26a3de1f8" candidate_uuid: "156308c205b841918853f294e89bd438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b546d49dac43838cff9bc9a6cfe056" is_pre_election: true
I20260923 17:22:21.714497 4278 raft_consensus.cc:2454] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 156308c205b841918853f294e89bd438 in term 0.
I20260923 17:22:21.714561 4411 raft_consensus.cc:2454] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 156308c205b841918853f294e89bd438 in term 0.
I20260923 17:22:21.715018 4484 leader_election.cc:302] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156308c205b841918853f294e89bd438, 3da3007573bc467fad9e2b65e6d76d36; no voters:
I20260923 17:22:21.715344 4634 raft_consensus.cc:2781] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:22:21.715459 4634 raft_consensus.cc:482] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:22:21.715588 4634 raft_consensus.cc:3037] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:21.717700 4634 raft_consensus.cc:504] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.718387 4634 leader_election.cc:288] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [CANDIDATE]: Term 1 election: Requested vote from peers 3da3007573bc467fad9e2b65e6d76d36 (127.28.72.193:37369), 00b546d49dac43838cff9bc9a6cfe056 (127.28.72.194:36227)
I20260923 17:22:21.718950 4411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a17313e24ac4770af725fa26a3de1f8" candidate_uuid: "156308c205b841918853f294e89bd438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b546d49dac43838cff9bc9a6cfe056"
I20260923 17:22:21.718902 4278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a17313e24ac4770af725fa26a3de1f8" candidate_uuid: "156308c205b841918853f294e89bd438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da3007573bc467fad9e2b65e6d76d36"
I20260923 17:22:21.719213 4278 raft_consensus.cc:3037] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:21.719234 4411 raft_consensus.cc:3037] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:21.722378 4411 raft_consensus.cc:2454] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 156308c205b841918853f294e89bd438 in term 1.
I20260923 17:22:21.722478 4278 raft_consensus.cc:2454] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 156308c205b841918853f294e89bd438 in term 1.
I20260923 17:22:21.722849 4485 leader_election.cc:302] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b546d49dac43838cff9bc9a6cfe056, 156308c205b841918853f294e89bd438; no voters:
I20260923 17:22:21.723349 4634 raft_consensus.cc:2781] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:21.723560 4634 raft_consensus.cc:686] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [term 1 LEADER]: Becoming Leader. State: Replica: 156308c205b841918853f294e89bd438, State: Running, Role: LEADER
I20260923 17:22:21.724084 4634 consensus_queue.cc:237] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:21.725885 4658 tablet_replica.cc:439] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438: TxnStatusTablet state changed. Reason: New leader 156308c205b841918853f294e89bd438. Latest consensus state: current_term: 1 leader_uuid: "156308c205b841918853f294e89bd438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } }
I20260923 17:22:21.726385 4658 tablet_replica.cc:442] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:22:21.727424 4115 catalog_manager.cc:5495] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 reported cstate change: term changed from 0 to 1, leader changed from <none> to 156308c205b841918853f294e89bd438 (127.28.72.195). New cstate: current_term: 1 leader_uuid: "156308c205b841918853f294e89bd438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } health_report { overall_health: HEALTHY } } }
I20260923 17:22:21.728667 4670 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:22:21.728896 4670 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:22:21.760134 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.752314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:40547 (local address 127.0.0.1:46200)
0923 17:22:21.752522 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:21.752544 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:21.752678 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:21.753258 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:21.753263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:21.753277 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:22:21.753477 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.753489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.755208 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.755214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.756814 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.756841 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.757965 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.757971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.758154 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.758235 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:22:21.759484 (+ 1249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:22:21.759542 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 17:22:21.759642 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 17:22:21.759805 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 17:22:21.760175 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.752461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.28.72.195:40547)
0923 17:22:21.752659 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:21.752667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:21.752690 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:22:21.752922 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:21.752930 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:21.753047 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:21.753225 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:21.753625 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.755026 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.756996 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.757771 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.757923 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.758483 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:21.759359 (+ 876us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:21.759525 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:21.759695 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 17:22:21.759794 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:22:21.765429 4411 raft_consensus.cc:1261] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Refusing update from remote peer 156308c205b841918853f294e89bd438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:21.765416 4278 raft_consensus.cc:1261] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Refusing update from remote peer 156308c205b841918853f294e89bd438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:21.766202 4634 consensus_queue.cc:1035] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:21.766919 4634 consensus_queue.cc:1035] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:21.780372 4633 tablet_replica.cc:439] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36: TxnStatusTablet state changed. Reason: New leader 156308c205b841918853f294e89bd438. Latest consensus state: current_term: 1 leader_uuid: "156308c205b841918853f294e89bd438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } }
I20260923 17:22:21.780745 4633 tablet_replica.cc:442] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:22:21.782603 4645 tablet_replica.cc:439] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056: TxnStatusTablet state changed. Reason: New leader 156308c205b841918853f294e89bd438. Latest consensus state: current_term: 1 leader_uuid: "156308c205b841918853f294e89bd438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } }
I20260923 17:22:21.782955 4645 tablet_replica.cc:442] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:22:21.786020 4658 tablet_replica.cc:439] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "156308c205b841918853f294e89bd438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } }
I20260923 17:22:21.786371 4658 tablet_replica.cc:442] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:22:21.790119 4634 tablet_replica.cc:439] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "156308c205b841918853f294e89bd438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } }
I20260923 17:22:21.790429 4634 tablet_replica.cc:442] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:22:21.796178 4645 tablet_replica.cc:439] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "156308c205b841918853f294e89bd438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } }
I20260923 17:22:21.796497 4645 tablet_replica.cc:442] T 6a17313e24ac4770af725fa26a3de1f8 P 00b546d49dac43838cff9bc9a6cfe056: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:22:21.797788 4633 tablet_replica.cc:439] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "156308c205b841918853f294e89bd438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } } }
I20260923 17:22:21.798153 4633 tablet_replica.cc:442] T 6a17313e24ac4770af725fa26a3de1f8 P 3da3007573bc467fad9e2b65e6d76d36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:22:21.802433 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.794826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.28.72.195:40547)
0923 17:22:21.795055 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:21.795061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:21.795081 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:22:21.795233 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:21.795238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:21.795339 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:21.795930 (+ 591us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:21.796094 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.797424 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.799754 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.800479 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.801186 (+ 707us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.801291 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:21.801431 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:21.801982 (+ 551us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:21.802032 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 17:22:21.802115 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:22:21.823477 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.815707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:51763 (local address 127.28.72.195:40547)
0923 17:22:21.815904 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:21.815921 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:21.816153 (+ 232us) server_negotiation.cc:407] Connection header received
0923 17:22:21.816415 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:21.816420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:21.816512 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:21.816569 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:21.817282 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.818535 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.821014 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.821704 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.821798 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.822619 (+ 821us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:21.822794 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:21.822852 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:21.823041 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 17:22:21.823150 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:22:21.824533 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.815432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:40547 (local address 127.28.72.195:51763)
0923 17:22:21.816058 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:21.816079 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:21.816273 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:21.816735 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:21.816740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:21.816751 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:22:21.817114 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.817126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.818700 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.818705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.820845 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:21.820864 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:21.822128 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:21.822160 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:21.822359 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:21.822441 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:22:21.822926 (+ 485us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:22:21.822934 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:22:21.823899 (+ 965us) client_negotiation.cc:240] Negotiation successful
0923 17:22:21.824088 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":526,"threads_started":1}
I20260923 17:22:21.838840 4278 raft_consensus.cc:1261] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Refusing update from remote peer 156308c205b841918853f294e89bd438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:21.838938 4411 raft_consensus.cc:3037] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:21.839643 4658 consensus_queue.cc:1035] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:21.843577 4411 raft_consensus.cc:1261] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Refusing update from remote peer 156308c205b841918853f294e89bd438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:21.844544 4658 consensus_queue.cc:1035] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:21.873093 4642 mvcc.cc:205] Tried to move back new op lower bound from 7332594244965990400 to 7332594243770871808. Current Snapshot: MvccSnapshot[applied={T|T < 7332594244965990400}]
I20260923 17:22:21.951485 4116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:21.951871 4116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:21.955564 4116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379: Sending DeleteTablet for 3 replicas of tablet 194712753ce84ae8b0ce5161724ea074
I20260923 17:22:21.956641 4253 tablet_service.cc:1514] Processing DeleteTablet for tablet 194712753ce84ae8b0ce5161724ea074 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:21 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:53688
I20260923 17:22:21.956650 4392 tablet_service.cc:1514] Processing DeleteTablet for tablet 194712753ce84ae8b0ce5161724ea074 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:21 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:38216
I20260923 17:22:21.956904 4650 tablet_replica.cc:330] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36: stopping tablet replica
I20260923 17:22:21.956931 4652 tablet_replica.cc:330] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056: stopping tablet replica
I20260923 17:22:21.957095 4650 raft_consensus.cc:2229] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:21.957126 4652 raft_consensus.cc:2229] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:21.957278 4650 raft_consensus.cc:2258] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:21.957350 4652 raft_consensus.cc:2258] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:21.958706 4650 ts_tablet_manager.cc:1905] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:21.958715 4652 ts_tablet_manager.cc:1905] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:21.966014 4650 ts_tablet_manager.cc:1918] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:22:21.966217 4650 log.cc:1196] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/194712753ce84ae8b0ce5161724ea074
I20260923 17:22:21.966662 4650 ts_tablet_manager.cc:1939] T 194712753ce84ae8b0ce5161724ea074 P 3da3007573bc467fad9e2b65e6d76d36: Deleting consensus metadata
I20260923 17:22:21.966773 4531 tablet_service.cc:1514] Processing DeleteTablet for tablet 194712753ce84ae8b0ce5161724ea074 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:21 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:46196
I20260923 17:22:21.967096 4651 tablet_replica.cc:330] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438: stopping tablet replica
I20260923 17:22:21.967345 4651 raft_consensus.cc:2229] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:22:21.967878 4651 raft_consensus.cc:2258] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:21.967923 4102 catalog_manager.cc:4841] TS 3da3007573bc467fad9e2b65e6d76d36 (127.28.72.193:37369): tablet 194712753ce84ae8b0ce5161724ea074 (table test-table [id=e3bdcc17b70c469ea1ec76bf5ac910b5]) successfully deleted
I20260923 17:22:21.969049 4651 ts_tablet_manager.cc:1905] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:21.974875 4652 ts_tablet_manager.cc:1918] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:22:21.975214 4652 log.cc:1196] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/194712753ce84ae8b0ce5161724ea074
I20260923 17:22:21.975625 4651 ts_tablet_manager.cc:1918] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:22:21.975772 4652 ts_tablet_manager.cc:1939] T 194712753ce84ae8b0ce5161724ea074 P 00b546d49dac43838cff9bc9a6cfe056: Deleting consensus metadata
I20260923 17:22:21.975786 4651 log.cc:1196] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/194712753ce84ae8b0ce5161724ea074
I20260923 17:22:21.976202 4651 ts_tablet_manager.cc:1939] T 194712753ce84ae8b0ce5161724ea074 P 156308c205b841918853f294e89bd438: Deleting consensus metadata
I20260923 17:22:21.977161 4103 catalog_manager.cc:4841] TS 00b546d49dac43838cff9bc9a6cfe056 (127.28.72.194:36227): tablet 194712753ce84ae8b0ce5161724ea074 (table test-table [id=e3bdcc17b70c469ea1ec76bf5ac910b5]) successfully deleted
I20260923 17:22:21.977427 4103 catalog_manager.cc:4841] TS 156308c205b841918853f294e89bd438 (127.28.72.195:40547): tablet 194712753ce84ae8b0ce5161724ea074 (table test-table [id=e3bdcc17b70c469ea1ec76bf5ac910b5]) successfully deleted
Sep 23 17:22:22 dist-test-slave-j0m4 krb5kdc[4071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:22:22 dist-test-slave-j0m4 krb5kdc[4071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:22 dist-test-slave-j0m4 krb5kdc[4071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:22:22.051640 4211 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:22.037070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.28.72.254:38211)
0923 17:22:22.037248 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:22.037254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:22.037803 (+ 549us) server_negotiation.cc:407] Connection header received
0923 17:22:22.038167 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:22.038174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:22.038265 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:22.038406 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:22.040141 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.041322 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.043656 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.044317 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.044451 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.051349 (+ 6898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53346: BlockingRecv error: recv got EOF from 127.0.0.1:53346 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 17:22:22 dist-test-slave-j0m4 krb5kdc[4071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM
I20260923 17:22:22.072495 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.055402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.28.72.254:38211)
0923 17:22:22.055621 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:22.055626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:22.055955 (+ 329us) server_negotiation.cc:407] Connection header received
0923 17:22:22.056105 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:22.056111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:22.056197 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:22.056326 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:22.057779 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.058897 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.060393 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.061050 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.061221 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.065487 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:22.065495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:22.065500 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:22.065545 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:22.069043 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:22.069778 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:22.069784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:22.069788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:22.069945 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:22.070334 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:22.070339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:22.070342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:22.070745 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:22.070989 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:22.071723 (+ 734us) server_negotiation.cc:299] Negotiation successful
0923 17:22:22.071998 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:22:22.076501 4116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:22:22.077169 4116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:22:22.084918 4253 tablet_service.cc:1467] Processing CreateTablet for tablet a1c7be8357f74615a9a0d3d83761bb96 (DEFAULT_TABLE table=test-table [id=13926fac73f74cf6bdda1e7ebf11e1c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:22.085455 4253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c7be8357f74615a9a0d3d83761bb96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:22.085330 4392 tablet_service.cc:1467] Processing CreateTablet for tablet a1c7be8357f74615a9a0d3d83761bb96 (DEFAULT_TABLE table=test-table [id=13926fac73f74cf6bdda1e7ebf11e1c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:22.085357 4531 tablet_service.cc:1467] Processing CreateTablet for tablet a1c7be8357f74615a9a0d3d83761bb96 (DEFAULT_TABLE table=test-table [id=13926fac73f74cf6bdda1e7ebf11e1c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:22.085767 4531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c7be8357f74615a9a0d3d83761bb96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:22.085775 4392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c7be8357f74615a9a0d3d83761bb96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:22.092540 4626 tablet_bootstrap.cc:492] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36: Bootstrap starting.
I20260923 17:22:22.092634 4628 tablet_bootstrap.cc:492] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438: Bootstrap starting.
I20260923 17:22:22.094097 4626 tablet_bootstrap.cc:654] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:22.094580 4628 tablet_bootstrap.cc:654] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:22.097127 4626 tablet_bootstrap.cc:492] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36: No bootstrap required, opened a new log
I20260923 17:22:22.097277 4626 ts_tablet_manager.cc:1397] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:22:22.097321 4627 tablet_bootstrap.cc:492] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056: Bootstrap starting.
I20260923 17:22:22.098088 4626 raft_consensus.cc:348] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.098364 4626 raft_consensus.cc:374] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:22.098448 4626 raft_consensus.cc:729] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da3007573bc467fad9e2b65e6d76d36, State: Initialized, Role: FOLLOWER
I20260923 17:22:22.098701 4626 consensus_queue.cc:260] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.099048 4627 tablet_bootstrap.cc:654] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:22.102885 4626 ts_tablet_manager.cc:1428] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260923 17:22:22.107029 4628 tablet_bootstrap.cc:492] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438: No bootstrap required, opened a new log
I20260923 17:22:22.107321 4628 ts_tablet_manager.cc:1397] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260923 17:22:22.107725 4627 tablet_bootstrap.cc:492] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056: No bootstrap required, opened a new log
I20260923 17:22:22.107873 4627 ts_tablet_manager.cc:1397] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 17:22:22.108407 4628 raft_consensus.cc:348] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.108901 4628 raft_consensus.cc:374] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:22.108711 4627 raft_consensus.cc:348] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.109042 4627 raft_consensus.cc:374] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:22.109148 4627 raft_consensus.cc:729] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b546d49dac43838cff9bc9a6cfe056, State: Initialized, Role: FOLLOWER
I20260923 17:22:22.109104 4628 raft_consensus.cc:729] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156308c205b841918853f294e89bd438, State: Initialized, Role: FOLLOWER
I20260923 17:22:22.109386 4627 consensus_queue.cc:260] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.109477 4628 consensus_queue.cc:260] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.109992 4627 ts_tablet_manager.cc:1428] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260923 17:22:22.110185 4628 ts_tablet_manager.cc:1428] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260923 17:22:22.122375 4645 raft_consensus.cc:482] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:22.122619 4645 raft_consensus.cc:504] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.123396 4645 leader_election.cc:288] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da3007573bc467fad9e2b65e6d76d36 (127.28.72.193:37369), 156308c205b841918853f294e89bd438 (127.28.72.195:40547)
I20260923 17:22:22.123957 4277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c7be8357f74615a9a0d3d83761bb96" candidate_uuid: "00b546d49dac43838cff9bc9a6cfe056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da3007573bc467fad9e2b65e6d76d36" is_pre_election: true
I20260923 17:22:22.124013 4551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c7be8357f74615a9a0d3d83761bb96" candidate_uuid: "00b546d49dac43838cff9bc9a6cfe056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156308c205b841918853f294e89bd438" is_pre_election: true
I20260923 17:22:22.124218 4277 raft_consensus.cc:2454] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b546d49dac43838cff9bc9a6cfe056 in term 0.
I20260923 17:22:22.124257 4551 raft_consensus.cc:2454] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b546d49dac43838cff9bc9a6cfe056 in term 0.
I20260923 17:22:22.124630 4345 leader_election.cc:302] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b546d49dac43838cff9bc9a6cfe056, 3da3007573bc467fad9e2b65e6d76d36; no voters:
I20260923 17:22:22.124913 4645 raft_consensus.cc:2781] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:22:22.125090 4645 raft_consensus.cc:482] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:22:22.125200 4645 raft_consensus.cc:3037] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:22.128384 4645 raft_consensus.cc:504] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.129062 4645 leader_election.cc:288] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [CANDIDATE]: Term 1 election: Requested vote from peers 3da3007573bc467fad9e2b65e6d76d36 (127.28.72.193:37369), 156308c205b841918853f294e89bd438 (127.28.72.195:40547)
I20260923 17:22:22.129563 4551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c7be8357f74615a9a0d3d83761bb96" candidate_uuid: "00b546d49dac43838cff9bc9a6cfe056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156308c205b841918853f294e89bd438"
I20260923 17:22:22.129559 4277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c7be8357f74615a9a0d3d83761bb96" candidate_uuid: "00b546d49dac43838cff9bc9a6cfe056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da3007573bc467fad9e2b65e6d76d36"
I20260923 17:22:22.129769 4277 raft_consensus.cc:3037] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:22.129802 4551 raft_consensus.cc:3037] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:22.132917 4551 raft_consensus.cc:2454] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b546d49dac43838cff9bc9a6cfe056 in term 1.
I20260923 17:22:22.132952 4277 raft_consensus.cc:2454] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b546d49dac43838cff9bc9a6cfe056 in term 1.
I20260923 17:22:22.133412 4346 leader_election.cc:302] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b546d49dac43838cff9bc9a6cfe056, 156308c205b841918853f294e89bd438; no voters:
I20260923 17:22:22.133749 4645 raft_consensus.cc:2781] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:22.134258 4645 raft_consensus.cc:686] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [term 1 LEADER]: Becoming Leader. State: Replica: 00b546d49dac43838cff9bc9a6cfe056, State: Running, Role: LEADER
I20260923 17:22:22.134582 4645 consensus_queue.cc:237] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.137357 4115 catalog_manager.cc:5495] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b546d49dac43838cff9bc9a6cfe056 (127.28.72.194). New cstate: current_term: 1 leader_uuid: "00b546d49dac43838cff9bc9a6cfe056" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } health_report { overall_health: UNKNOWN } } }
I20260923 17:22:22.156049 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.149548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.28.72.194:36227)
0923 17:22:22.150182 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:22.150188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:22.150210 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:22:22.150281 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:22.150289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:22.150381 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:22.150498 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:22.150977 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.152172 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.154217 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.154659 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.154817 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.155253 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:22.155365 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:22.155463 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:22.155607 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 17:22:22.155682 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":389,"threads_started":1}
I20260923 17:22:22.158932 4277 raft_consensus.cc:1261] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Refusing update from remote peer 00b546d49dac43838cff9bc9a6cfe056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:22.159605 4551 raft_consensus.cc:1261] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Refusing update from remote peer 00b546d49dac43838cff9bc9a6cfe056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:22.159564 4698 consensus_queue.cc:1035] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:22.160163 4698 consensus_queue.cc:1035] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:22.171310 4641 mvcc.cc:205] Tried to move back new op lower bound from 7332594246276345856 to 7332594246186278912. Current Snapshot: MvccSnapshot[applied={T|T < 7332594246276345856 or (T in {7332594246276345856})}]
I20260923 17:22:22.175886 4114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:22.176173 4114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:22.179921 4114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379: Sending DeleteTablet for 3 replicas of tablet a1c7be8357f74615a9a0d3d83761bb96
I20260923 17:22:22.180706 4253 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c7be8357f74615a9a0d3d83761bb96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:22 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:53688
I20260923 17:22:22.181344 4531 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c7be8357f74615a9a0d3d83761bb96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:22 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:46196
I20260923 17:22:22.181509 4650 tablet_replica.cc:330] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36: stopping tablet replica
I20260923 17:22:22.181684 4650 raft_consensus.cc:2229] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:22.181803 4651 tablet_replica.cc:330] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438: stopping tablet replica
I20260923 17:22:22.181954 4650 raft_consensus.cc:2258] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:22.182030 4651 raft_consensus.cc:2229] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:22.182313 4651 raft_consensus.cc:2258] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:22.182317 4530 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c7be8357f74615a9a0d3d83761bb96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:22 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:46196
I20260923 17:22:22.182958 4392 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c7be8357f74615a9a0d3d83761bb96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:22 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:38216
I20260923 17:22:22.182818 4116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 17:22:22.183275 4652 tablet_replica.cc:330] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056: stopping tablet replica
I20260923 17:22:22.183460 4652 raft_consensus.cc:2229] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:22:22.183638 4650 ts_tablet_manager.cc:1905] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:22.183801 4652 raft_consensus.cc:2258] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 17:22:22.183822 4116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:22:22.184908 4652 ts_tablet_manager.cc:1905] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:22.185869 4651 ts_tablet_manager.cc:1905] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:22.191488 4650 ts_tablet_manager.cc:1918] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:22.191742 4650 log.cc:1196] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/a1c7be8357f74615a9a0d3d83761bb96
I20260923 17:22:22.192323 4650 ts_tablet_manager.cc:1939] T a1c7be8357f74615a9a0d3d83761bb96 P 3da3007573bc467fad9e2b65e6d76d36: Deleting consensus metadata
I20260923 17:22:22.193522 4102 catalog_manager.cc:4841] TS 3da3007573bc467fad9e2b65e6d76d36 (127.28.72.193:37369): tablet a1c7be8357f74615a9a0d3d83761bb96 (table test-table [id=13926fac73f74cf6bdda1e7ebf11e1c2]) successfully deleted
I20260923 17:22:22.195102 4253 tablet_service.cc:1467] Processing CreateTablet for tablet d4802181117a4c13b67be39dda8cf8d6 (DEFAULT_TABLE table=test-table [id=ee26cc7ed9e44be29ff7815bd801e2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:22.195623 4253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4802181117a4c13b67be39dda8cf8d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:22.195725 4530 tablet_service.cc:1467] Processing CreateTablet for tablet d4802181117a4c13b67be39dda8cf8d6 (DEFAULT_TABLE table=test-table [id=ee26cc7ed9e44be29ff7815bd801e2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:22.196213 4530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4802181117a4c13b67be39dda8cf8d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:22.196617 4651 ts_tablet_manager.cc:1918] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:22.196785 4651 log.cc:1196] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/a1c7be8357f74615a9a0d3d83761bb96
I20260923 17:22:22.198853 4651 ts_tablet_manager.cc:1939] T a1c7be8357f74615a9a0d3d83761bb96 P 156308c205b841918853f294e89bd438: Deleting consensus metadata
I20260923 17:22:22.200017 4392 tablet_service.cc:1467] Processing CreateTablet for tablet d4802181117a4c13b67be39dda8cf8d6 (DEFAULT_TABLE table=test-table [id=ee26cc7ed9e44be29ff7815bd801e2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:22.200479 4392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4802181117a4c13b67be39dda8cf8d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:22.202643 4626 tablet_bootstrap.cc:492] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36: Bootstrap starting.
I20260923 17:22:22.204437 4626 tablet_bootstrap.cc:654] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:22.206616 4627 tablet_bootstrap.cc:492] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056: Bootstrap starting.
I20260923 17:22:22.206660 4626 tablet_bootstrap.cc:492] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36: No bootstrap required, opened a new log
I20260923 17:22:22.206967 4626 ts_tablet_manager.cc:1397] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260923 17:22:22.207352 4103 catalog_manager.cc:4841] TS 156308c205b841918853f294e89bd438 (127.28.72.195:40547): tablet a1c7be8357f74615a9a0d3d83761bb96 (table test-table [id=13926fac73f74cf6bdda1e7ebf11e1c2]) successfully deleted
W20260923 17:22:22.207808 4103 catalog_manager.cc:4808] TS 156308c205b841918853f294e89bd438 (127.28.72.195:40547): delete failed for tablet a1c7be8357f74615a9a0d3d83761bb96 because the tablet was not found. No further retry: Not found: Tablet not found: a1c7be8357f74615a9a0d3d83761bb96
I20260923 17:22:22.208273 4627 tablet_bootstrap.cc:654] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:22.208393 4626 raft_consensus.cc:348] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.208709 4626 raft_consensus.cc:374] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:22.208834 4626 raft_consensus.cc:729] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da3007573bc467fad9e2b65e6d76d36, State: Initialized, Role: FOLLOWER
I20260923 17:22:22.209725 4626 consensus_queue.cc:260] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.210376 4626 ts_tablet_manager.cc:1428] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:22:22.217998 4628 tablet_bootstrap.cc:492] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438: Bootstrap starting.
I20260923 17:22:22.219480 4627 tablet_bootstrap.cc:492] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056: No bootstrap required, opened a new log
I20260923 17:22:22.219735 4627 ts_tablet_manager.cc:1397] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260923 17:22:22.219784 4652 ts_tablet_manager.cc:1918] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:22.219933 4652 log.cc:1196] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/a1c7be8357f74615a9a0d3d83761bb96
I20260923 17:22:22.219991 4628 tablet_bootstrap.cc:654] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:22.220516 4652 ts_tablet_manager.cc:1939] T a1c7be8357f74615a9a0d3d83761bb96 P 00b546d49dac43838cff9bc9a6cfe056: Deleting consensus metadata
I20260923 17:22:22.220649 4627 raft_consensus.cc:348] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.220997 4627 raft_consensus.cc:374] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:22.221093 4627 raft_consensus.cc:729] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b546d49dac43838cff9bc9a6cfe056, State: Initialized, Role: FOLLOWER
I20260923 17:22:22.221365 4627 consensus_queue.cc:260] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.221689 4103 catalog_manager.cc:4841] TS 00b546d49dac43838cff9bc9a6cfe056 (127.28.72.194:36227): tablet a1c7be8357f74615a9a0d3d83761bb96 (table test-table [id=13926fac73f74cf6bdda1e7ebf11e1c2]) successfully deleted
I20260923 17:22:22.222877 4627 ts_tablet_manager.cc:1428] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:22:22.225713 4628 tablet_bootstrap.cc:492] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438: No bootstrap required, opened a new log
I20260923 17:22:22.225915 4628 ts_tablet_manager.cc:1397] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260923 17:22:22.226857 4628 raft_consensus.cc:348] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.227093 4628 raft_consensus.cc:374] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:22.227228 4628 raft_consensus.cc:729] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156308c205b841918853f294e89bd438, State: Initialized, Role: FOLLOWER
I20260923 17:22:22.227469 4628 consensus_queue.cc:260] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.228440 4628 ts_tablet_manager.cc:1428] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 17:22:22.358243 4633 raft_consensus.cc:482] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:22.358572 4633 raft_consensus.cc:504] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.360210 4633 leader_election.cc:288] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b546d49dac43838cff9bc9a6cfe056 (127.28.72.194:36227), 156308c205b841918853f294e89bd438 (127.28.72.195:40547)
I20260923 17:22:22.370887 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.360520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:48603 (local address 127.28.72.194:36227)
0923 17:22:22.360732 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:22.360739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:22.361163 (+ 424us) server_negotiation.cc:407] Connection header received
0923 17:22:22.361483 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:22.361493 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:22.361595 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:22.361672 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:22.362537 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.364045 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.367490 (+ 3445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.369224 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.369405 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.369520 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:22.370249 (+ 729us) server_negotiation.cc:299] Negotiation successful
0923 17:22:22.370464 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:22:22.371976 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.360394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:36227 (local address 127.28.72.193:48603)
0923 17:22:22.361068 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:22.361090 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:22.361276 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:22.361906 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:22.361914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:22.361951 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:22.362321 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:22.362336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.364282 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.364290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:22.367233 (+ 2943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:22.367254 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.369713 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.369722 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:22.369999 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.370055 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 17:22:22.371257 (+ 1202us) client_negotiation.cc:240] Negotiation successful
0923 17:22:22.371484 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":384,"threads_started":1}
I20260923 17:22:22.373646 4412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4802181117a4c13b67be39dda8cf8d6" candidate_uuid: "3da3007573bc467fad9e2b65e6d76d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b546d49dac43838cff9bc9a6cfe056" is_pre_election: true
I20260923 17:22:22.373915 4412 raft_consensus.cc:2454] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da3007573bc467fad9e2b65e6d76d36 in term 0.
I20260923 17:22:22.374475 4206 leader_election.cc:302] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b546d49dac43838cff9bc9a6cfe056, 3da3007573bc467fad9e2b65e6d76d36; no voters:
I20260923 17:22:22.374927 4633 raft_consensus.cc:2781] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:22:22.375054 4633 raft_consensus.cc:482] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:22:22.375195 4633 raft_consensus.cc:3037] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:22.377503 4633 raft_consensus.cc:504] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.378283 4633 leader_election.cc:288] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [CANDIDATE]: Term 1 election: Requested vote from peers 00b546d49dac43838cff9bc9a6cfe056 (127.28.72.194:36227), 156308c205b841918853f294e89bd438 (127.28.72.195:40547)
I20260923 17:22:22.378841 4412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4802181117a4c13b67be39dda8cf8d6" candidate_uuid: "3da3007573bc467fad9e2b65e6d76d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b546d49dac43838cff9bc9a6cfe056"
I20260923 17:22:22.379079 4412 raft_consensus.cc:3037] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:22.380198 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.361203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:40547 (local address 127.28.72.193:40825)
0923 17:22:22.372307 (+ 11104us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:22.372335 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:22.372494 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:22.373764 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:22.373769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:22.373797 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:22.373980 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:22.373990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.375312 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.375319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:22.377416 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:22.377432 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.379367 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.379386 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:22.379572 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.379606 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:22:22.379710 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 17:22:22.379860 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10960,"thread_start_us":11607,"threads_started":1}
I20260923 17:22:22.380230 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.361326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:40825 (local address 127.28.72.195:40547)
0923 17:22:22.373455 (+ 12129us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:22.373462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:22.373483 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:22:22.373563 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:22.373567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:22.373626 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:22.373748 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:22.374122 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.375140 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.377640 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.378941 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.379170 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.379262 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:22.379764 (+ 502us) server_negotiation.cc:299] Negotiation successful
0923 17:22:22.379916 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11962,"thread_start_us":11814,"threads_started":1}
I20260923 17:22:22.381456 4550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4802181117a4c13b67be39dda8cf8d6" candidate_uuid: "3da3007573bc467fad9e2b65e6d76d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156308c205b841918853f294e89bd438" is_pre_election: true
I20260923 17:22:22.381564 4551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4802181117a4c13b67be39dda8cf8d6" candidate_uuid: "3da3007573bc467fad9e2b65e6d76d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156308c205b841918853f294e89bd438"
I20260923 17:22:22.381763 4550 raft_consensus.cc:2454] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da3007573bc467fad9e2b65e6d76d36 in term 0.
I20260923 17:22:22.382138 4412 raft_consensus.cc:2454] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da3007573bc467fad9e2b65e6d76d36 in term 1.
I20260923 17:22:22.382570 4206 leader_election.cc:302] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b546d49dac43838cff9bc9a6cfe056, 3da3007573bc467fad9e2b65e6d76d36; no voters:
I20260923 17:22:22.382903 4633 raft_consensus.cc:2781] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:22.383579 4633 raft_consensus.cc:686] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 LEADER]: Becoming Leader. State: Replica: 3da3007573bc467fad9e2b65e6d76d36, State: Running, Role: LEADER
I20260923 17:22:22.383993 4633 consensus_queue.cc:237] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } }
I20260923 17:22:22.387730 4115 catalog_manager.cc:5495] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3da3007573bc467fad9e2b65e6d76d36 (127.28.72.193). New cstate: current_term: 1 leader_uuid: "3da3007573bc467fad9e2b65e6d76d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da3007573bc467fad9e2b65e6d76d36" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 37369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 } health_report { overall_health: UNKNOWN } } }
I20260923 17:22:22.420670 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.407007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.28.72.193:37369)
0923 17:22:22.413989 (+ 6982us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:22.413996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:22.414029 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:22:22.414149 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:22.414154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:22.414263 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:22.414496 (+ 233us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:22.414947 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.416188 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.417736 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.418393 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.418525 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.419050 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:22.419915 (+ 865us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:22.420085 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:22.420208 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 17:22:22.420295 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6823,"thread_start_us":7701,"threads_started":1}
I20260923 17:22:22.438733 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.431735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:50109 (local address 127.28.72.195:40547)
0923 17:22:22.431933 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:22.431940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:22.432589 (+ 649us) server_negotiation.cc:407] Connection header received
0923 17:22:22.432814 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:22.432818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:22.432887 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:22.432976 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:22.433566 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.434498 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.436654 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.437154 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.437348 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.438036 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:22.438143 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:22.438215 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:22.438449 (+ 234us) server_negotiation.cc:299] Negotiation successful
0923 17:22:22.438521 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:22:22.438897 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.431619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:40547 (local address 127.28.72.193:50109)
0923 17:22:22.432474 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:22.432494 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:22.432648 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:22.433013 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:22.433019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:22.433034 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:22:22.433395 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:22.433407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.434659 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.434665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:22.436476 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:22.436493 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.437481 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.437493 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:22.437679 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.437769 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:22:22.438243 (+ 474us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:22:22.438257 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 17:22:22.438354 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 17:22:22.438495 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":454,"threads_started":1}
I20260923 17:22:22.448879 4412 raft_consensus.cc:1261] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Refusing update from remote peer 3da3007573bc467fad9e2b65e6d76d36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:22.448942 4551 raft_consensus.cc:3037] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:22.449782 4710 consensus_queue.cc:1035] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b546d49dac43838cff9bc9a6cfe056" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:22.452632 4551 raft_consensus.cc:1261] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Refusing update from remote peer 3da3007573bc467fad9e2b65e6d76d36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:22.453383 4633 consensus_queue.cc:1035] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "156308c205b841918853f294e89bd438" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:22.513619 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.499979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:52201 (local address 127.28.72.193:37369)
0923 17:22:22.500168 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:22.500175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:22.507855 (+ 7680us) server_negotiation.cc:407] Connection header received
0923 17:22:22.507943 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:22.507948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:22.508041 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:22.508139 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:22.508629 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.509738 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.511497 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.512124 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.512255 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.512765 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:22.512939 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:22.513015 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:22.513261 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 17:22:22.513349 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:22:22.513731 4721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.499564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:37369 (local address 127.28.72.195:52201)
0923 17:22:22.507719 (+ 8155us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:22.507739 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:22.507852 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:22.508161 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:22.508164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:22.508174 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:22:22.508437 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:22.508446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.509889 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.509893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:22.511349 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:22.511363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.512336 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.512342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:22.512528 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.512581 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:22:22.513039 (+ 458us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:22:22.513050 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 17:22:22.513211 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 17:22:22.513381 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7996,"thread_start_us":7833,"threads_started":1}
I20260923 17:22:22.552981 4115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:22.553333 4115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:22.557947 4115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e584ffce34a34c61b58a05e30d566379: Sending DeleteTablet for 3 replicas of tablet d4802181117a4c13b67be39dda8cf8d6
I20260923 17:22:22.559099 4392 tablet_service.cc:1514] Processing DeleteTablet for tablet d4802181117a4c13b67be39dda8cf8d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:22 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:38216
I20260923 17:22:22.559156 4530 tablet_service.cc:1514] Processing DeleteTablet for tablet d4802181117a4c13b67be39dda8cf8d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:22 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:46196
I20260923 17:22:22.559459 4651 tablet_replica.cc:330] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438: stopping tablet replica
I20260923 17:22:22.559473 4652 tablet_replica.cc:330] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056: stopping tablet replica
I20260923 17:22:22.559676 4652 raft_consensus.cc:2229] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:22.559686 4651 raft_consensus.cc:2229] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:22.559871 4652 raft_consensus.cc:2258] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:22.559890 4651 raft_consensus.cc:2258] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:22.561266 4651 ts_tablet_manager.cc:1905] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:22.561607 4652 ts_tablet_manager.cc:1905] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:22.569172 4651 ts_tablet_manager.cc:1918] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:22:22.569199 4253 tablet_service.cc:1514] Processing DeleteTablet for tablet d4802181117a4c13b67be39dda8cf8d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:22 UTC) from {username='oryx', principal='oryx/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:53688
I20260923 17:22:22.569475 4651 log.cc:1196] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/d4802181117a4c13b67be39dda8cf8d6
I20260923 17:22:22.569556 4650 tablet_replica.cc:330] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36: stopping tablet replica
I20260923 17:22:22.569832 4650 raft_consensus.cc:2229] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:22:22.570039 4651 ts_tablet_manager.cc:1939] T d4802181117a4c13b67be39dda8cf8d6 P 156308c205b841918853f294e89bd438: Deleting consensus metadata
I20260923 17:22:22.570396 4650 raft_consensus.cc:2258] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:22.571398 4103 catalog_manager.cc:4841] TS 156308c205b841918853f294e89bd438 (127.28.72.195:40547): tablet d4802181117a4c13b67be39dda8cf8d6 (table test-table [id=ee26cc7ed9e44be29ff7815bd801e2ef]) successfully deleted
I20260923 17:22:22.572345 4650 ts_tablet_manager.cc:1905] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:22.575589 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 4187
I20260923 17:22:22.582288 4652 ts_tablet_manager.cc:1918] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:22:22.582950 4650 ts_tablet_manager.cc:1918] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:22:22.583184 4650 log.cc:1196] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/d4802181117a4c13b67be39dda8cf8d6
I20260923 17:22:22.583864 4650 ts_tablet_manager.cc:1939] T d4802181117a4c13b67be39dda8cf8d6 P 3da3007573bc467fad9e2b65e6d76d36: Deleting consensus metadata
I20260923 17:22:22.585160 4102 catalog_manager.cc:4841] TS 3da3007573bc467fad9e2b65e6d76d36 (127.28.72.193:37369): tablet d4802181117a4c13b67be39dda8cf8d6 (table test-table [id=ee26cc7ed9e44be29ff7815bd801e2ef]) successfully deleted
I20260923 17:22:22.585256 4652 log.cc:1196] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/d4802181117a4c13b67be39dda8cf8d6
I20260923 17:22:22.586236 4652 ts_tablet_manager.cc:1939] T d4802181117a4c13b67be39dda8cf8d6 P 00b546d49dac43838cff9bc9a6cfe056: Deleting consensus metadata
I20260923 17:22:22.587509 4103 catalog_manager.cc:4841] TS 00b546d49dac43838cff9bc9a6cfe056 (127.28.72.194:36227): tablet d4802181117a4c13b67be39dda8cf8d6 (table test-table [id=ee26cc7ed9e44be29ff7815bd801e2ef]) successfully deleted
Sep 23 17:22:22 dist-test-slave-j0m4 krb5kdc[4071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.72.193@KRBTEST.COM
I20260923 17:22:22.606140 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.576626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53700 (local address 127.28.72.193:37369)
0923 17:22:22.576830 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:22.576836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:22.585958 (+ 9122us) server_negotiation.cc:407] Connection header received
0923 17:22:22.586305 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:22.586316 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:22.586433 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:22.586520 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:22.588620 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.589828 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.591768 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.592475 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.592620 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.598196 (+ 5576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:22.598215 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:22.598239 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:22.598285 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:22.602604 (+ 4319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:22.603631 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:22.603646 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:22.603650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:22.603828 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:22.604301 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:22.604306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:22.604310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:22.604794 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:22.605175 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:22.605447 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 17:22:22.605698 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:22:22.607033 4318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:22.736573 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 4187
I20260923 17:22:22.764369 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 4327
Sep 23 17:22:22 dist-test-slave-j0m4 krb5kdc[4071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.72.194@KRBTEST.COM
I20260923 17:22:22.781847 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.765242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.28.72.194:36227)
0923 17:22:22.765429 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:22.765435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:22.765456 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:22:22.765611 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:22.765616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:22.765694 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:22.765797 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:22.767048 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.768208 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.769485 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.770143 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.770273 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.774653 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:22.774672 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:22.774692 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:22.774742 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:22.778708 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:22.779545 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:22.779554 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:22.779559 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:22.779717 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:22.780130 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:22.780135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:22.780139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:22.780580 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:22.780874 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:22.781174 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 17:22:22.781440 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:22:22.782668 4457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:22.909571 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 4327
I20260923 17:22:22.932721 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 4466
I20260923 17:22:22.939671 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.938488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:36227 (local address 127.28.72.195:38351)
0923 17:22:22.939295 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:22.939500 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.194:36227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":447,"threads_started":1}
W20260923 17:22:22.940017 4485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.72.194:36227: connect: Connection refused (error 111)
I20260923 17:22:22.941660 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.941210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:36227 (local address 127.28.72.195:59005)
0923 17:22:22.941443 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:22.941530 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.194:36227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260923 17:22:22.942037 4485 consensus_peers.cc:487] T 6a17313e24ac4770af725fa26a3de1f8 P 156308c205b841918853f294e89bd438 -> Peer 00b546d49dac43838cff9bc9a6cfe056 (127.28.72.194:36227): Couldn't send request to peer 00b546d49dac43838cff9bc9a6cfe056. Status: Network error: Client connection negotiation failed: client connection to 127.28.72.194:36227: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 23 17:22:22 dist-test-slave-j0m4 krb5kdc[4071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.72.195@KRBTEST.COM
I20260923 17:22:22.950150 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.933691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.28.72.195:40547)
0923 17:22:22.933903 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:22.933910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:22.933928 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:22:22.933991 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:22.933995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:22.934055 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:22.934166 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:22.935577 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.936288 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.937688 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:22.938326 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:22.938477 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:22.943905 (+ 5428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:22.943926 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:22.943942 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:22.943991 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:22.946853 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:22.947878 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:22.947889 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:22.947893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:22.948044 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:22.948545 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:22.948548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:22.948551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:22.948953 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:22.949259 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:22.949578 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 17:22:22.949806 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:22:22.950795 4596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:23.097620 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 4466
I20260923 17:22:23.123512 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 4084
I20260923 17:22:23.124809 4156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:23.241951 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 4084
2026-09-23T17:22:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6779 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:22:23 dist-test-slave-j0m4 krb5kdc[4733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:22:23 dist-test-slave-j0m4 krb5kdc[4733](info): set up 2 sockets
Sep 23 17:22:23 dist-test-slave-j0m4 krb5kdc[4733](info): commencing operation
krb5kdc: starting...
W20260923 17:22:24.392506 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:22:24 dist-test-slave-j0m4 krb5kdc[4733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:22:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:22:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:22:24.644290 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:41147
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:34937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:41147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:24.989532 4746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:24.989913 4746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:24.990069 4746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:24.997888 4746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:24.998061 4746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:24.998174 4746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:24.998277 4746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:24.998377 4746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:25.008898 4746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:34937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:41147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:41147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:25.010303 4746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:25.011785 4746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:25.024380 4752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:25.024308 4751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:25.025082 4754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:25.026063 4746 server_base.cc:1034] running on GCE node
I20260923 17:22:25.026973 4746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:25.028524 4746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:25.029793 4746 hybrid_clock.cc:648] HybridClock initialized: now 1790184145029744 us; error 42 us; skew 500 ppm
I20260923 17:22:25.030234 4746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T17:22:25Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1861 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:22:25 dist-test-slave-j0m4 krb5kdc[4760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:22:25 dist-test-slave-j0m4 krb5kdc[4760](info): set up 2 sockets
Sep 23 17:22:25 dist-test-slave-j0m4 krb5kdc[4760](info): commencing operation
krb5kdc: starting...
W20260923 17:22:26.240229 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:22:26 dist-test-slave-j0m4 krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:22:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:22:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:22:26.492440 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:36239
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:36239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:26.829744 4773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:26.830054 4773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:26.830219 4773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:26.838020 4773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:26.838158 4773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:26.838254 4773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:26.838335 4773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:26.838493 4773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:26.848932 4773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:36239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:36239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:26.850306 4773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:26.851728 4773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:26.864583 4778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:26.864637 4779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:26.865015 4781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:26.865393 4773 server_base.cc:1034] running on GCE node
I20260923 17:22:26.866255 4773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:26.867954 4773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:26.869182 4773 hybrid_clock.cc:648] HybridClock initialized: now 1790184146869139 us; error 36 us; skew 500 ppm
I20260923 17:22:26.869673 4773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:26 dist-test-slave-j0m4 krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184146, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:26.881285 4773 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:22:26.882926 4773 webserver.cc:460] Webserver started at http://127.28.72.254:37309/ using document root <none> and password file <none>
I20260923 17:22:26.883738 4773 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:26.883860 4773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:26.884128 4773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:26.887485 4773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "31a0a74c66944b159845772f8ece1ce5"
format_stamp: "Formatted at 2026-09-23 17:22:26 on dist-test-slave-j0m4"
server_key: "858ac702ea83c742b14a7941bdc90411"
server_key_iv: "1a737b4279a4626c5808d1cca17018bc"
server_key_version: "encryptionkey@0"
I20260923 17:22:26.888238 4773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "31a0a74c66944b159845772f8ece1ce5"
format_stamp: "Formatted at 2026-09-23 17:22:26 on dist-test-slave-j0m4"
server_key: "858ac702ea83c742b14a7941bdc90411"
server_key_iv: "1a737b4279a4626c5808d1cca17018bc"
server_key_version: "encryptionkey@0"
I20260923 17:22:26.894001 4773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:22:26.898206 4788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:26.899855 4773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:26.900060 4773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "31a0a74c66944b159845772f8ece1ce5"
format_stamp: "Formatted at 2026-09-23 17:22:26 on dist-test-slave-j0m4"
server_key: "858ac702ea83c742b14a7941bdc90411"
server_key_iv: "1a737b4279a4626c5808d1cca17018bc"
server_key_version: "encryptionkey@0"
I20260923 17:22:26.900276 4773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:26.930109 4773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:26.934443 4773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:26.934749 4773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:26.955235 4773 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:36239
I20260923 17:22:26.955228 4839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:36239 every 8 connection(s)
I20260923 17:22:26.956679 4773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:26.960809 4840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:26.966730 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 4773
I20260923 17:22:26.967005 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:22:26.967712 28963 external_mini_cluster.cc:1428] Setting key afa0ed28c0a9ed689b60536b97e32e3b
I20260923 17:22:26.972493 4840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5: Bootstrap starting.
I20260923 17:22:26.976570 4840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:26.977825 4840 log.cc:824] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:26.981091 4840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5: No bootstrap required, opened a new log
Sep 23 17:22:26 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:22:26.989951 4840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a0a74c66944b159845772f8ece1ce5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36239 } }
I20260923 17:22:26.990420 4840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:26.990563 4840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a0a74c66944b159845772f8ece1ce5, State: Initialized, Role: FOLLOWER
I20260923 17:22:26.991353 4840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a0a74c66944b159845772f8ece1ce5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36239 } }
I20260923 17:22:26.991658 4840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:26.991844 4840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:26.992044 4840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:26.992305 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:26.970136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57252 (local address 127.28.72.254:36239)
0923 17:22:26.971006 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:26.971019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:26.971064 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:22:26.972973 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:26.972998 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:26.973443 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:26.973797 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:26.975848 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:26.977280 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:26.979099 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:26.979843 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:26.980045 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:26.984399 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:26.984408 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:26.984432 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:26.984475 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:26.987778 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:26.988566 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:26.988577 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:26.988583 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:26.988756 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:26.989154 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:26.989160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:26.989163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:26.989725 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:26.990025 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:26.990501 (+ 476us) server_negotiation.cc:299] Negotiation successful
0923 17:22:26.990978 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":406,"threads_started":1}
I20260923 17:22:26.995397 4840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a0a74c66944b159845772f8ece1ce5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36239 } }
I20260923 17:22:26.995961 4840 leader_election.cc:302] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31a0a74c66944b159845772f8ece1ce5; no voters:
I20260923 17:22:26.996590 4840 leader_election.cc:288] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:22:26.996693 4845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:26.997517 4845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 1 LEADER]: Becoming Leader. State: Replica: 31a0a74c66944b159845772f8ece1ce5, State: Running, Role: LEADER
I20260923 17:22:26.998034 4845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a0a74c66944b159845772f8ece1ce5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36239 } }
I20260923 17:22:26.998865 4840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:27.001075 4846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31a0a74c66944b159845772f8ece1ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a0a74c66944b159845772f8ece1ce5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36239 } } }
I20260923 17:22:27.001626 4846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:27.001657 4847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31a0a74c66944b159845772f8ece1ce5. Latest consensus state: current_term: 1 leader_uuid: "31a0a74c66944b159845772f8ece1ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a0a74c66944b159845772f8ece1ce5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36239 } } }
I20260923 17:22:27.002276 4847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:27.003556 4851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:27.010046 4851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:27.019794 4851 catalog_manager.cc:1347] Generated new cluster ID: 9e17ff80515a4512b79a9e4fd1f74518
I20260923 17:22:27.019971 4851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:27.050500 4851 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:22:27.052043 4851 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:27.060207 4851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5: Generated new TSK 0
I20260923 17:22:27.061153 4851 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:22:27.128986 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:36239
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:27.496665 4866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:27.496961 4866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:27.497149 4866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:27.505055 4866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:27.505167 4866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:27.505405 4866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:22:27.515420 4866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:36239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:27.516768 4866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:27.518108 4866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:27.531857 4871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:27.531893 4874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:27.531816 4872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:27.533042 4866 server_base.cc:1034] running on GCE node
I20260923 17:22:27.533754 4866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:27.534876 4866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:27.536154 4866 hybrid_clock.cc:648] HybridClock initialized: now 1790184147536070 us; error 90 us; skew 500 ppm
I20260923 17:22:27.536669 4866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:27 dist-test-slave-j0m4 krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:27.540297 4866 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:22:27.542086 4866 webserver.cc:460] Webserver started at http://127.28.72.193:37889/ using document root <none> and password file <none>
I20260923 17:22:27.542886 4866 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:27.543035 4866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:27.543404 4866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:27.546459 4866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "f23be3c902e7448387c498e9e3aee0e8"
format_stamp: "Formatted at 2026-09-23 17:22:27 on dist-test-slave-j0m4"
server_key: "6f318029199d5ebbf6d6d435916f1f73"
server_key_iv: "a61c901efff0774e4819bb08cf915f2e"
server_key_version: "encryptionkey@0"
I20260923 17:22:27.547401 4866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "f23be3c902e7448387c498e9e3aee0e8"
format_stamp: "Formatted at 2026-09-23 17:22:27 on dist-test-slave-j0m4"
server_key: "6f318029199d5ebbf6d6d435916f1f73"
server_key_iv: "a61c901efff0774e4819bb08cf915f2e"
server_key_version: "encryptionkey@0"
I20260923 17:22:27.553007 4866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:22:27.556874 4881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:27.558465 4866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:22:27.558669 4866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "f23be3c902e7448387c498e9e3aee0e8"
format_stamp: "Formatted at 2026-09-23 17:22:27 on dist-test-slave-j0m4"
server_key: "6f318029199d5ebbf6d6d435916f1f73"
server_key_iv: "a61c901efff0774e4819bb08cf915f2e"
server_key_version: "encryptionkey@0"
I20260923 17:22:27.558915 4866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:27.587776 4866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:27.592407 4866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:27.592666 4866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:27.593616 4866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:27.594990 4866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:27.595062 4866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:27.595230 4866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:27.595284 4866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:27.629926 4866 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:44225
I20260923 17:22:27.629987 4993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:44225 every 8 connection(s)
I20260923 17:22:27.631379 4866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:22:27.637058 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 4866
I20260923 17:22:27.637390 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:22:27.638355 28963 external_mini_cluster.cc:1428] Setting key 451baa0333b77491dcfcfe1fbb453559
Sep 23 17:22:27 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:22:27.655279 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:27.634177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:43733 (local address 127.28.72.254:36239)
0923 17:22:27.634680 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:27.634685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:27.635758 (+ 1073us) server_negotiation.cc:407] Connection header received
0923 17:22:27.637737 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:27.637744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:27.637890 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:27.638026 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:27.639998 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:27.641210 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:27.642930 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:27.643697 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:27.643853 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:27.648086 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:27.648095 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:27.648103 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:27.648150 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:27.651377 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:27.652498 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:27.652503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:27.652505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:27.652639 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:27.653220 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:27.653225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:27.653227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:27.653580 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:27.653711 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:27.654746 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0923 17:22:27.654929 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":276,"threads_started":1}
I20260923 17:22:27.656847 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:27.634828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:43733)
0923 17:22:27.635592 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:27.635632 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:27.637369 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:27.638248 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:27.638258 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:27.638865 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:27.639749 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:27.639779 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:27.641394 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:27.641399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:27.642763 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:27.642776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:27.644031 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:27.644036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:27.644259 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:27.645083 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:27.645107 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:27.647805 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:27.651575 (+ 3770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:27.651588 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:27.651613 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:27.652315 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:27.652766 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:27.652774 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:27.652780 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:27.653068 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:27.653726 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:27.653738 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:27.654259 (+ 521us) client_negotiation.cc:769] Sending connection context
0923 17:22:27.654600 (+ 341us) client_negotiation.cc:240] Negotiation successful
0923 17:22:27.655276 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":334,"threads_started":1}
I20260923 17:22:27.658900 4994 heartbeater.cc:347] Connected to a master server at 127.28.72.254:36239
I20260923 17:22:27.659288 4994 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:27.660141 4994 heartbeater.cc:511] Master 127.28.72.254:36239 requested a full tablet report, sending...
I20260923 17:22:27.662788 4805 ts_manager.cc:194] Registered new tserver with Master: f23be3c902e7448387c498e9e3aee0e8 (127.28.72.193:44225)
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
I20260923 17:22:27.664846 4805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:43733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:22:27.695179 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:36239
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:28.034039 5000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:28.034381 5000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:28.034541 5000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:28.042095 5000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:28.042230 5000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:28.042424 5000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:22:28.053006 5000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:36239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:28.054483 5000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:28.055872 5000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:28.070340 5005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:28.070369 5006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:28.070478 5008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:28.071686 5000 server_base.cc:1034] running on GCE node
I20260923 17:22:28.072482 5000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:28.073629 5000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:28.074877 5000 hybrid_clock.cc:648] HybridClock initialized: now 1790184148074804 us; error 72 us; skew 500 ppm
I20260923 17:22:28.075287 5000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:28 dist-test-slave-j0m4 krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184148, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:28.079074 5000 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:22:28.080993 5000 webserver.cc:460] Webserver started at http://127.28.72.194:37051/ using document root <none> and password file <none>
I20260923 17:22:28.081802 5000 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:28.081969 5000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:28.082319 5000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:28.085669 5000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "abcf4b9db031406e84c9f0d31180b91b"
format_stamp: "Formatted at 2026-09-23 17:22:28 on dist-test-slave-j0m4"
server_key: "dbd0473378fc8927c4c1a8ee201b084d"
server_key_iv: "21fc753eab6f2661bed9ffa11296ce53"
server_key_version: "encryptionkey@0"
I20260923 17:22:28.086562 5000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "abcf4b9db031406e84c9f0d31180b91b"
format_stamp: "Formatted at 2026-09-23 17:22:28 on dist-test-slave-j0m4"
server_key: "dbd0473378fc8927c4c1a8ee201b084d"
server_key_iv: "21fc753eab6f2661bed9ffa11296ce53"
server_key_version: "encryptionkey@0"
I20260923 17:22:28.093140 5000 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.001s
I20260923 17:22:28.097783 5015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:28.099763 5000 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 17:22:28.099934 5000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "abcf4b9db031406e84c9f0d31180b91b"
format_stamp: "Formatted at 2026-09-23 17:22:28 on dist-test-slave-j0m4"
server_key: "dbd0473378fc8927c4c1a8ee201b084d"
server_key_iv: "21fc753eab6f2661bed9ffa11296ce53"
server_key_version: "encryptionkey@0"
I20260923 17:22:28.100126 5000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:28.118045 5000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:28.122611 5000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:28.122880 5000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:28.123971 5000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:28.125416 5000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:28.125514 5000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:28.125602 5000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:28.125646 5000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:28.160969 5000 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:40755
I20260923 17:22:28.161028 5127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:40755 every 8 connection(s)
I20260923 17:22:28.162361 5000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:22:28.169206 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 5000
I20260923 17:22:28.169589 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:22:28.170107 28963 external_mini_cluster.cc:1428] Setting key f1fa6d1952d6a30deeeb82c40a312267
Sep 23 17:22:28 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184148, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:22:28.184876 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:28.165171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:41757 (local address 127.28.72.254:36239)
0923 17:22:28.165638 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:28.165643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:28.166743 (+ 1100us) server_negotiation.cc:407] Connection header received
0923 17:22:28.168744 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:28.168749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:28.168836 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:28.168967 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:28.170783 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:28.171687 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:28.173416 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:28.173873 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:28.174011 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:28.178147 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:28.178153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:28.178156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:28.178193 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:28.181313 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:28.182203 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:28.182209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:28.182215 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:28.182359 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:28.182842 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:28.182850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:28.182853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:28.183229 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:28.183420 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:28.184177 (+ 757us) server_negotiation.cc:299] Negotiation successful
0923 17:22:28.184435 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":236,"threads_started":1}
I20260923 17:22:28.185945 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:28.165770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:41757)
0923 17:22:28.166550 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:28.166595 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:28.168445 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:28.169221 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:28.169230 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:28.169942 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:28.170596 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:28.170615 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:28.171865 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:28.171869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:28.173265 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:28.173277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:28.174107 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:28.174115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:28.174360 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:28.175314 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:28.175345 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:28.177833 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:28.181521 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:28.181532 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:28.181560 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:28.182049 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:28.182497 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:28.182502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:28.182507 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:28.182718 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:28.183369 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:28.183380 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:28.183783 (+ 403us) client_negotiation.cc:769] Sending connection context
0923 17:22:28.184091 (+ 308us) client_negotiation.cc:240] Negotiation successful
0923 17:22:28.184689 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":301,"threads_started":1}
I20260923 17:22:28.187677 5128 heartbeater.cc:347] Connected to a master server at 127.28.72.254:36239
I20260923 17:22:28.188069 5128 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:28.189106 5128 heartbeater.cc:511] Master 127.28.72.254:36239 requested a full tablet report, sending...
I20260923 17:22:28.191267 4805 ts_manager.cc:194] Registered new tserver with Master: abcf4b9db031406e84c9f0d31180b91b (127.28.72.194:40755)
I20260923 17:22:28.192153 4805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:41757
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:22:28.227710 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:36239
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:28.574069 5134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:28.574491 5134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:28.574688 5134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:28.583243 5134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:28.583443 5134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:28.583645 5134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:22:28.594187 5134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:36239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:28.595728 5134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:28.597179 5134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:28.612671 5140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:28.613138 5139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:28.613211 5142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:28.613916 5134 server_base.cc:1034] running on GCE node
I20260923 17:22:28.614657 5134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:28.615895 5134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:28.617175 5134 hybrid_clock.cc:648] HybridClock initialized: now 1790184148617116 us; error 50 us; skew 500 ppm
I20260923 17:22:28.617596 5134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:28 dist-test-slave-j0m4 krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184148, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:28.622133 5134 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:22:28.624082 5134 webserver.cc:460] Webserver started at http://127.28.72.195:33033/ using document root <none> and password file <none>
I20260923 17:22:28.625095 5134 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:28.625278 5134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:28.625627 5134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:28.628818 5134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "2c7e3eef0adb48c1bb9ddac19107ad21"
format_stamp: "Formatted at 2026-09-23 17:22:28 on dist-test-slave-j0m4"
server_key: "1351d512ebb3afea2992d288423c6859"
server_key_iv: "d92c00b0f04afa93e16f629c6904c0cf"
server_key_version: "encryptionkey@0"
I20260923 17:22:28.629750 5134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "2c7e3eef0adb48c1bb9ddac19107ad21"
format_stamp: "Formatted at 2026-09-23 17:22:28 on dist-test-slave-j0m4"
server_key: "1351d512ebb3afea2992d288423c6859"
server_key_iv: "d92c00b0f04afa93e16f629c6904c0cf"
server_key_version: "encryptionkey@0"
I20260923 17:22:28.636543 5134 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 17:22:28.640918 5149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:28.642576 5134 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:22:28.642797 5134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "2c7e3eef0adb48c1bb9ddac19107ad21"
format_stamp: "Formatted at 2026-09-23 17:22:28 on dist-test-slave-j0m4"
server_key: "1351d512ebb3afea2992d288423c6859"
server_key_iv: "d92c00b0f04afa93e16f629c6904c0cf"
server_key_version: "encryptionkey@0"
I20260923 17:22:28.643045 5134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:28.657130 5134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:28.662281 5134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:28.662652 5134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:28.663872 5134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:28.665587 5134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:28.665704 5134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:28.665865 5134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:28.665938 5134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:28.668339 4994 heartbeater.cc:503] Master 127.28.72.254:36239 was elected leader, sending a full tablet report...
I20260923 17:22:28.703977 5134 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:39859
I20260923 17:22:28.703998 5261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:39859 every 8 connection(s)
I20260923 17:22:28.705729 5134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:22:28.712253 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 5134
I20260923 17:22:28.712517 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:22:28.713064 28963 external_mini_cluster.cc:1428] Setting key 397bff38c19985c003b8f8a268164273
Sep 23 17:22:28 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184148, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:22:28.729283 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:28.708595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:37325 (local address 127.28.72.254:36239)
0923 17:22:28.709087 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:28.709095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:28.710192 (+ 1097us) server_negotiation.cc:407] Connection header received
0923 17:22:28.712245 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:28.712250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:28.712356 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:28.712491 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:28.714710 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:28.715718 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:28.717734 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:28.718159 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:28.718306 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:28.722297 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:28.722305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:28.722310 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:28.722367 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:28.725266 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:28.726262 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:28.726267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:28.726270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:28.726407 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:28.726894 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:28.726901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:28.726904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:28.727350 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:28.727501 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:28.728658 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0923 17:22:28.728918 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":222,"threads_started":1}
I20260923 17:22:28.730371 5265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:28.709229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:37325)
0923 17:22:28.710015 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:28.710056 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:28.711922 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:28.712820 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:28.712835 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:28.713657 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:28.714464 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:28.714517 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:28.715923 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:28.715927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:28.717549 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:28.717566 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:28.718442 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:28.718452 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:28.718707 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:28.719556 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:28.719585 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:28.722002 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:28.725543 (+ 3541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:28.725552 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:28.725568 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:28.726107 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:28.726538 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:28.726541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:28.726543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:28.726740 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:28.727526 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:28.727537 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:28.728153 (+ 616us) client_negotiation.cc:769] Sending connection context
0923 17:22:28.728474 (+ 321us) client_negotiation.cc:240] Negotiation successful
0923 17:22:28.729012 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":317,"threads_started":1}
I20260923 17:22:28.731889 5262 heartbeater.cc:347] Connected to a master server at 127.28.72.254:36239
I20260923 17:22:28.732159 5262 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:28.732841 5262 heartbeater.cc:511] Master 127.28.72.254:36239 requested a full tablet report, sending...
I20260923 17:22:28.734426 4805 ts_manager.cc:194] Registered new tserver with Master: 2c7e3eef0adb48c1bb9ddac19107ad21 (127.28.72.195:39859)
I20260923 17:22:28.735350 4805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:37325
I20260923 17:22:28.742789 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.193.keytab.
Entry for principal oryx/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.193.keytab.
WARNING: no policy specified for oryx/127.28.72.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.72.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.194.keytab.
Entry for principal oryx/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.194.keytab.
WARNING: no policy specified for oryx/127.28.72.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.72.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.195.keytab.
Entry for principal oryx/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.195.keytab.
I20260923 17:22:28.903055 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 4866
Sep 23 17:22:28 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:22:28.920663 5273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:28.904468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37444 (local address 127.28.72.193:44225)
0923 17:22:28.905097 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:28.905104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:28.905369 (+ 265us) server_negotiation.cc:407] Connection header received
0923 17:22:28.905595 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:28.905604 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:28.905820 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:28.905992 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:28.907426 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:28.908558 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:28.909857 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:28.910469 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:28.910586 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:28.914720 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:28.914734 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:28.914749 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:28.914788 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:28.917655 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:28.918424 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:28.918434 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:28.918438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:28.918583 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:28.918965 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:28.918970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:28.918975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:28.919391 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:28.919698 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:28.920022 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 17:22:28.920298 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":322,"threads_started":1}
I20260923 17:22:28.922036 4989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:29.009030 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 4866
I20260923 17:22:29.026497 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 5000
Sep 23 17:22:29 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:22:29.043769 5275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:29.027605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.28.72.194:40755)
0923 17:22:29.028112 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:29.028119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:29.028144 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:22:29.028208 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:29.028214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:29.028444 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:29.028581 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:29.029857 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:29.030960 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:29.032412 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:29.033028 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:29.033138 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:29.037617 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:29.037633 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:29.037649 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:29.037689 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:29.040836 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:29.041672 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:29.041680 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:29.041682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:29.041820 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:29.042192 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:29.042195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:29.042197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:29.042560 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:29.042792 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:29.043159 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 17:22:29.043372 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":256,"threads_started":1}
I20260923 17:22:29.045243 5123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:29.132501 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 5000
I20260923 17:22:29.149250 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 5134
Sep 23 17:22:29 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:22:29.166903 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:29.150390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.28.72.195:39859)
0923 17:22:29.150954 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:29.150961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:29.150992 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:22:29.151070 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:29.151077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:29.151346 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:29.151524 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:29.152924 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:29.153920 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:29.155328 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:29.155923 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:29.156094 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:29.160845 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:29.160860 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:29.160877 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:29.160918 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:29.163978 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:29.164700 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:29.164708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:29.164710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:29.164844 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:29.165234 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:29.165240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:29.165244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:29.165634 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:29.165946 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:29.166249 (+ 303us) server_negotiation.cc:299] Negotiation successful
0923 17:22:29.166515 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":241,"threads_started":1}
I20260923 17:22:29.168212 5257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:29.252574 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 5134
I20260923 17:22:29.268821 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 4773
I20260923 17:22:29.270051 4835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:29.373855 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 4773
I20260923 17:22:29.396935 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:36239
--webserver_interface=127.28.72.254
--webserver_port=37309
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:36239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:29.738448 5280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:29.738832 5280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:29.739007 5280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:29.746712 5280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:29.746887 5280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:29.747012 5280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:29.747109 5280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:29.747217 5280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:29.757496 5280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:36239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:36239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=37309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:29.759002 5280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:29.760406 5280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:29.772737 5288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:29.772845 5285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:29.772682 5286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:29.773903 5280 server_base.cc:1034] running on GCE node
I20260923 17:22:29.774816 5280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:29.776499 5280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:29.777742 5280 hybrid_clock.cc:648] HybridClock initialized: now 1790184149777704 us; error 74 us; skew 500 ppm
I20260923 17:22:29.778120 5280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:29 dist-test-slave-j0m4 krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184149, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:29.781483 5280 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:22:29.783174 5280 webserver.cc:460] Webserver started at http://127.28.72.254:37309/ using document root <none> and password file <none>
I20260923 17:22:29.783923 5280 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:29.784088 5280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:29.790082 5280 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:22:29.793358 5295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:29.795027 5280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:22:29.795300 5280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "31a0a74c66944b159845772f8ece1ce5"
format_stamp: "Formatted at 2026-09-23 17:22:26 on dist-test-slave-j0m4"
server_key: "858ac702ea83c742b14a7941bdc90411"
server_key_iv: "1a737b4279a4626c5808d1cca17018bc"
server_key_version: "encryptionkey@0"
I20260923 17:22:29.796331 5280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:29.816044 5280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:29.820778 5280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:29.821110 5280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:29.839663 5280 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:36239
I20260923 17:22:29.839681 5346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:36239 every 8 connection(s)
I20260923 17:22:29.841341 5280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:29.848963 5347 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:22:29.850967 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 5280
I20260923 17:22:29.851986 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:44225
--local_ip_for_outbound_sockets=127.28.72.193
--tserver_master_addrs=127.28.72.254:36239
--webserver_port=37889
--webserver_interface=127.28.72.193
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.72.193
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:22:29.853948 5347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5: Bootstrap starting.
I20260923 17:22:29.875428 5347 log.cc:824] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:29.884626 5347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:22:29.885186 5347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5: Bootstrap complete.
I20260923 17:22:29.898357 5347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a0a74c66944b159845772f8ece1ce5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36239 } }
I20260923 17:22:29.899268 5347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a0a74c66944b159845772f8ece1ce5, State: Initialized, Role: FOLLOWER
I20260923 17:22:29.899998 5347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a0a74c66944b159845772f8ece1ce5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36239 } }
I20260923 17:22:29.900266 5347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:29.900416 5347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:29.900614 5347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:22:29.904708 5347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a0a74c66944b159845772f8ece1ce5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36239 } }
I20260923 17:22:29.905236 5347 leader_election.cc:302] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31a0a74c66944b159845772f8ece1ce5; no voters:
I20260923 17:22:29.905864 5347 leader_election.cc:288] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:22:29.905998 5351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:22:29.907157 5351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [term 2 LEADER]: Becoming Leader. State: Replica: 31a0a74c66944b159845772f8ece1ce5, State: Running, Role: LEADER
I20260923 17:22:29.907740 5351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a0a74c66944b159845772f8ece1ce5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36239 } }
I20260923 17:22:29.908198 5347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:29.910534 5353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31a0a74c66944b159845772f8ece1ce5. Latest consensus state: current_term: 2 leader_uuid: "31a0a74c66944b159845772f8ece1ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a0a74c66944b159845772f8ece1ce5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36239 } } }
I20260923 17:22:29.911022 5353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:29.911777 5352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "31a0a74c66944b159845772f8ece1ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a0a74c66944b159845772f8ece1ce5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 36239 } } }
I20260923 17:22:29.912171 5352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:29.915716 5357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:29.921500 5357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:29.922950 5357 catalog_manager.cc:1259] Loaded cluster ID: 9e17ff80515a4512b79a9e4fd1f74518
I20260923 17:22:29.923058 5357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:29.927511 5357 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:29.929020 5357 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 31a0a74c66944b159845772f8ece1ce5: Loaded TSK: 0
I20260923 17:22:29.930749 5357 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:22:30.234586 5349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:30.234906 5349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:30.235069 5349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:30.242705 5349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:30.242823 5349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:30.243023 5349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:22:30.253538 5349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.193.keytab
--principal=oryx/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:44225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=37889
--tserver_master_addrs=127.28.72.254:36239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:30.254993 5349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:30.256379 5349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:30.271807 5373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:30.271907 5374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:30.273149 5376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:30.273236 5349 server_base.cc:1034] running on GCE node
I20260923 17:22:30.274067 5349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:30.275273 5349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:30.276571 5349 hybrid_clock.cc:648] HybridClock initialized: now 1790184150276497 us; error 62 us; skew 500 ppm
I20260923 17:22:30.277019 5349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:30 dist-test-slave-j0m4 krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184150, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:30.281114 5349 init.cc:377] Logged in from keytab as oryx/127.28.72.193@KRBTEST.COM (short username oryx)
I20260923 17:22:30.282919 5349 webserver.cc:460] Webserver started at http://127.28.72.193:37889/ using document root <none> and password file <none>
I20260923 17:22:30.283767 5349 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:30.283934 5349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:30.289830 5349 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 17:22:30.292856 5383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:30.294734 5349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:22:30.294912 5349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "f23be3c902e7448387c498e9e3aee0e8"
format_stamp: "Formatted at 2026-09-23 17:22:27 on dist-test-slave-j0m4"
server_key: "6f318029199d5ebbf6d6d435916f1f73"
server_key_iv: "a61c901efff0774e4819bb08cf915f2e"
server_key_version: "encryptionkey@0"
I20260923 17:22:30.296038 5349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:30.314823 5349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:30.319797 5349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:30.320081 5349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:30.321265 5349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:30.322656 5349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:30.322751 5349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:30.322902 5349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:30.323001 5349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:30.362092 5349 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:44225
I20260923 17:22:30.362298 5495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:44225 every 8 connection(s)
I20260923 17:22:30.363732 5349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:22:30.370726 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 5349
I20260923 17:22:30.371548 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:40755
--local_ip_for_outbound_sockets=127.28.72.194
--tserver_master_addrs=127.28.72.254:36239
--webserver_port=37051
--webserver_interface=127.28.72.194
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.72.194
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:22:30 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:30 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:22:30.718796 5500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:30.719166 5500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:30.719344 5500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:30.726830 5500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:30.726980 5500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:30.727211 5500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:22:30.737591 5500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.194.keytab
--principal=oryx/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:40755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=37051
--tserver_master_addrs=127.28.72.254:36239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:30.739029 5500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:30.740404 5500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:30.757112 5507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:30.757458 5509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:30.758198 5506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:30.758916 5500 server_base.cc:1034] running on GCE node
I20260923 17:22:30.759683 5500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:30.760797 5500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:30.764185 5500 hybrid_clock.cc:648] HybridClock initialized: now 1790184150764080 us; error 100 us; skew 500 ppm
I20260923 17:22:30.764599 5500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:30 dist-test-slave-j0m4 krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184150, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:30.768857 5500 init.cc:377] Logged in from keytab as oryx/127.28.72.194@KRBTEST.COM (short username oryx)
I20260923 17:22:30.770955 5500 webserver.cc:460] Webserver started at http://127.28.72.194:37051/ using document root <none> and password file <none>
I20260923 17:22:30.771843 5500 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:30.772030 5500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:30.778985 5500 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:22:30.782840 5516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:30.784976 5500 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:22:30.785194 5500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "abcf4b9db031406e84c9f0d31180b91b"
format_stamp: "Formatted at 2026-09-23 17:22:28 on dist-test-slave-j0m4"
server_key: "dbd0473378fc8927c4c1a8ee201b084d"
server_key_iv: "21fc753eab6f2661bed9ffa11296ce53"
server_key_version: "encryptionkey@0"
I20260923 17:22:30.786296 5500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:30.808037 5500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:30.814172 5500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:30.814543 5500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:30.815861 5500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:30.817567 5500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:30.817716 5500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:30.817868 5500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:30.817966 5500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:30.872334 5500 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:40755
I20260923 17:22:30.872378 5628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:40755 every 8 connection(s)
I20260923 17:22:30.874164 5500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:22:30.880402 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 5500
I20260923 17:22:30.881067 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:39859
--local_ip_for_outbound_sockets=127.28.72.195
--tserver_master_addrs=127.28.72.254:36239
--webserver_port=33033
--webserver_interface=127.28.72.195
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.72.195
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:22:30 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:30 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:22:31.256675 5633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:31.257112 5633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:31.257282 5633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:31.268509 5633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:31.268662 5633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:31.268848 5633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:22:31.279563 5633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/krb5kdc/oryx_127.28.72.195.keytab
--principal=oryx/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:39859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=33033
--tserver_master_addrs=127.28.72.254:36239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:31.281060 5633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:31.282768 5633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:31.301244 5640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:31.303750 5639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:31.309266 5642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:31.310201 5633 server_base.cc:1034] running on GCE node
I20260923 17:22:31.311437 5633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:31.313375 5633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:31.316869 5633 hybrid_clock.cc:648] HybridClock initialized: now 1790184151316734 us; error 98 us; skew 500 ppm
I20260923 17:22:31.317543 5633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:31 dist-test-slave-j0m4 krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184151, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:31.324676 5633 init.cc:377] Logged in from keytab as oryx/127.28.72.195@KRBTEST.COM (short username oryx)
I20260923 17:22:31.328907 5633 webserver.cc:460] Webserver started at http://127.28.72.195:33033/ using document root <none> and password file <none>
I20260923 17:22:31.330555 5633 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:31.330922 5633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:31.343331 5633 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.007s sys 0.000s
I20260923 17:22:31.349843 5649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:31.353048 5633 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.003s
I20260923 17:22:31.353392 5633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "2c7e3eef0adb48c1bb9ddac19107ad21"
format_stamp: "Formatted at 2026-09-23 17:22:28 on dist-test-slave-j0m4"
server_key: "1351d512ebb3afea2992d288423c6859"
server_key_iv: "d92c00b0f04afa93e16f629c6904c0cf"
server_key_version: "encryptionkey@0"
I20260923 17:22:31.355787 5633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:31.428264 5633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:31.437423 5633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:31.437907 5633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:31.439888 5633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:31.442763 5633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:31.442962 5633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:31.443156 5633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:31.443279 5633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260923 17:22:31.512548 5633 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:39859
I20260923 17:22:31.512645 5761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:39859 every 8 connection(s)
I20260923 17:22:31.515534 5633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:22:31.522588 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 5633
Sep 23 17:22:31 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:31 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:31.584547 5765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:31.524694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.28.72.254:36239)
0923 17:22:31.545037 (+ 20343us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:31.545047 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:31.545079 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:22:31.545170 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:31.545179 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:31.545308 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:31.545483 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:31.551817 (+ 6334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:31.553745 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:31.556875 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:31.557882 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:31.558092 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:31.563817 (+ 5725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:31.563836 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:31.563872 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:31.563957 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:31.574993 (+ 11036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:31.576557 (+ 1564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:31.576573 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:31.576591 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:31.577379 (+ 788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:31.578089 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:31.578098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:31.578104 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:31.579078 (+ 974us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:31.579701 (+ 623us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:31.582056 (+ 2355us) server_negotiation.cc:299] Negotiation successful
0923 17:22:31.582828 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20153,"thread_start_us":15412,"threads_started":1}
W20260923 17:22:31.866670 5499 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.487s user 0.000s sys 0.018s
W20260923 17:22:31.869272 5498 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:30.366891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:55713 (local address 127.28.72.254:36239)
0923 17:22:30.367678 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:30.367701 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:30.368041 (+ 340us) server_negotiation.cc:407] Connection header received
0923 17:22:30.370396 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:30.370410 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:30.370808 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:30.371228 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:30.373575 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:30.374854 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:30.377645 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:30.378305 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:30.378540 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:31.868708 (+1490168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:55713: BlockingRecv error: recv got EOF from 127.28.72.193:55713 (error 108)
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":325,"threads_started":1}
I20260923 17:22:31.872920 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:30.366913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:55713)
0923 17:22:30.367850 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:30.367899 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:30.369804 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:30.371447 (+ 1643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:30.371459 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:30.372344 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:30.373231 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:30.373263 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:30.375057 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:30.375066 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:30.377473 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:30.377494 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:30.378583 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:30.378590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:30.378799 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:30.379584 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:30.379609 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:31.870911 (+1491302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":436,"threads_started":1}
W20260923 17:22:31.875020 5496 heartbeater.cc:644] Failed to heartbeat to 127.28.72.254:36239 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.72.254:36239: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Sep 23 17:22:31 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:31 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:31.920209 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:31.876160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:40239)
0923 17:22:31.878697 (+ 2537us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:31.878726 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:31.879753 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:31.880786 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:31.880794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:31.881512 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:31.881846 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:31.881875 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:31.892881 (+ 11006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:31.892892 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:31.901260 (+ 8368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:31.901299 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:31.903652 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:31.903663 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:31.903967 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:31.912693 (+ 8726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:31.912740 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:31.919673 (+ 6933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2364,"thread_start_us":30570,"threads_started":1}
W20260923 17:22:31.920475 5498 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:31.876716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:40239 (local address 127.28.72.254:36239)
0923 17:22:31.877110 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:31.877119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:31.878851 (+ 1732us) server_negotiation.cc:407] Connection header received
0923 17:22:31.880188 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:31.880197 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:31.880322 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:31.880427 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:31.885501 (+ 5074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:31.891874 (+ 6373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:31.901857 (+ 9983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:31.902877 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:31.903127 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:31.920095 (+ 16968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:40239: BlockingRecv error: recv got EOF from 127.28.72.193:40239 (error 108)
Metrics: {"server-negotiator.queue_time_us":218}
Sep 23 17:22:31 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:31 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:31.939903 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:31.921552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:53707)
0923 17:22:31.921880 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:31.921913 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:31.922108 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:31.922756 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:31.922764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:31.923703 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:31.924184 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:31.924205 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:31.926585 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:31.926596 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:31.929042 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:31.929071 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:31.930766 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:31.930777 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:31.931103 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:31.932256 (+ 1153us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:31.932294 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:31.939389 (+ 7095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260923 17:22:31.940733 5496 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:22:31.941797 5498 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:31.922051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:53707 (local address 127.28.72.254:36239)
0923 17:22:31.922288 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:31.922298 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:31.922325 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:22:31.922412 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:31.922420 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:31.922528 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:31.922676 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:31.924468 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:31.926286 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:31.929289 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:31.930451 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:31.930644 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:31.941369 (+ 10725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:53707: BlockingRecv error: recv got EOF from 127.28.72.193:53707 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 17:22:32.242774 5632 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.346s user 0.005s sys 0.016s
W20260923 17:22:32.244796 5631 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:30.878077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:52873 (local address 127.28.72.254:36239)
0923 17:22:30.878765 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:30.878773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:30.880636 (+ 1863us) server_negotiation.cc:407] Connection header received
0923 17:22:30.884510 (+ 3874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:30.884521 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:30.884651 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:30.884801 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:30.888294 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:30.889822 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:30.894084 (+ 4262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:30.894972 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:30.895206 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:32.244543 (+1349337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:52873: BlockingRecv error: recv got EOF from 127.28.72.194:52873 (error 108)
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":289,"threads_started":1}
I20260923 17:22:32.246115 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:30.878776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:52873)
0923 17:22:30.880020 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:30.880094 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:30.883939 (+ 3845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:30.885289 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:30.885311 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:30.886569 (+ 1258us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:30.887924 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:30.887969 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:30.890089 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:30.890098 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:30.893747 (+ 3649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:30.893781 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:30.895418 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:30.895429 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:30.895939 (+ 510us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:30.897105 (+ 1166us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:30.897162 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:32.244278 (+1347116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":484,"threads_started":1}
W20260923 17:22:32.247403 5629 heartbeater.cc:644] Failed to heartbeat to 127.28.72.254:36239 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.72.254:36239: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Sep 23 17:22:32 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:32 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:32.262647 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:32.248404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:57539)
0923 17:22:32.248803 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:32.248833 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:32.249127 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:32.249618 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:32.249626 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:32.250365 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:32.250701 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:32.250719 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.252010 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.252021 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:32.254586 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:32.254613 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.255545 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.255556 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:32.255873 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:32.256846 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:32.256887 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:32.262311 (+ 5424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210}
W20260923 17:22:32.262744 5631 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:32.248490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:57539 (local address 127.28.72.254:36239)
0923 17:22:32.248664 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:32.248669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:32.248957 (+ 288us) server_negotiation.cc:407] Connection header received
0923 17:22:32.249298 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:32.249301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:32.249441 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:32.249524 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:32.250908 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.251828 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.254812 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.255299 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.255407 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:32.262500 (+ 7093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:57539: BlockingRecv error: recv got EOF from 127.28.72.194:57539 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 17:22:32 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:32 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:32.272879 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:32.263575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:57995)
0923 17:22:32.263767 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:32.263794 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:32.263903 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:32.264289 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:32.264294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:32.264626 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:32.264774 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:32.264782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.265781 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.265787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:32.266745 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:32.266756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.267534 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.267540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:32.267729 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:32.268582 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:32.268601 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:32.272596 (+ 3995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260923 17:22:32.272970 5631 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:32.263813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:57995 (local address 127.28.72.254:36239)
0923 17:22:32.264017 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:32.264023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:32.264037 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:22:32.264088 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:32.264091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:32.264152 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:32.264256 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:32.264915 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.265635 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.266892 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.267383 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.267521 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:32.272700 (+ 5179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:57995: BlockingRecv error: recv got EOF from 127.28.72.194:57995 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 17:22:32.273317 5629 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:22:32.793504 5764 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.217s user 0.001s sys 0.024s
W20260923 17:22:32.794978 5767 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:31.523672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:36957 (local address 127.28.72.254:36239)
0923 17:22:31.540093 (+ 16421us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:31.540104 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:31.549108 (+ 9004us) server_negotiation.cc:407] Connection header received
0923 17:22:31.555739 (+ 6631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:31.555750 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:31.555897 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:31.556017 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:31.561894 (+ 5877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:31.563818 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:31.572454 (+ 8636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:31.573586 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:31.573830 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:32.794662 (+1220832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:36957: BlockingRecv error: recv got EOF from 127.28.72.195:36957 (error 108)
Metrics: {"server-negotiator.queue_time_us":16195,"thread_start_us":20994,"threads_started":1}
I20260923 17:22:32.796003 5764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:31.521119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:36957)
0923 17:22:31.548112 (+ 26993us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:31.548212 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:31.555065 (+ 6853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:31.558744 (+ 3679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:31.558768 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:31.560104 (+ 1336us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:31.561511 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:31.561559 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:31.565657 (+ 4098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:31.565717 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:31.570673 (+ 4956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:31.570712 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:31.574470 (+ 3758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:31.574483 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:31.575001 (+ 518us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:31.576273 (+ 1272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:31.576323 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:32.794470 (+1218147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26427,"thread_start_us":26045,"threads_started":1}
W20260923 17:22:32.797014 5762 heartbeater.cc:644] Failed to heartbeat to 127.28.72.254:36239 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.72.254:36239: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Sep 23 17:22:32 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:32 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:32.807411 5764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:32.797721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:34213)
0923 17:22:32.797924 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:32.797947 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:32.798072 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:32.798403 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:32.798407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:32.798797 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:32.798966 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:32.798975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.800156 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.800163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:32.801317 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:32.801330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.802106 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.802114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:32.802273 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:32.802972 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:32.802992 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:32.807076 (+ 4084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260923 17:22:32.807770 5767 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:32.797876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:34213 (local address 127.28.72.254:36239)
0923 17:22:32.798075 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:32.798081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:32.798102 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:22:32.798197 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:32.798202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:32.798271 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:32.798375 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:32.799154 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.800011 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.801483 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.801900 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.802038 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:32.807440 (+ 5402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:34213: BlockingRecv error: recv got EOF from 127.28.72.195:34213 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 23 17:22:32 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:32 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:32.817409 5764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:32.808170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:59835)
0923 17:22:32.808364 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:32.808387 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:32.808513 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:32.808825 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:32.808828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:32.809156 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:32.809388 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:32.809398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.810391 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.810395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:32.811525 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:32.811538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.812268 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.812273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:32.812457 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:32.813046 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:32.813065 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:32.816984 (+ 3919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 17:22:32.817612 5767 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:32.808261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:59835 (local address 127.28.72.254:36239)
0923 17:22:32.808544 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:32.808555 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:32.808572 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:22:32.808619 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:32.808622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:32.808688 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:32.808775 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:32.809544 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.810225 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.811674 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.812111 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.812218 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:32.817283 (+ 5065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:59835: BlockingRecv error: recv got EOF from 127.28.72.195:59835 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260923 17:22:32.817800 5762 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 17:22:32 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:32 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:32.951767 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:32.941401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:57237)
0923 17:22:32.941938 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:32.941957 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:32.942079 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:32.942545 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:32.942550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:32.943001 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:32.943260 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:32.943277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.944865 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.944869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:32.945901 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:32.945915 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.946647 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.946652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:32.946771 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:32.947430 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:32.947448 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:32.951527 (+ 4079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":311,"threads_started":1}
W20260923 17:22:32.952087 5767 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:32.941630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:57237 (local address 127.28.72.254:36239)
0923 17:22:32.941807 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:32.941812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:32.942078 (+ 266us) server_negotiation.cc:407] Connection header received
0923 17:22:32.942219 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:32.942225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:32.942370 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:32.942507 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:32.943504 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.944674 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.946073 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:32.946501 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:32.946611 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:32.951764 (+ 5153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:57237: BlockingRecv error: recv got EOF from 127.28.72.193:57237 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 17:22:33 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:33 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:33.284292 5771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:33.273929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:56227)
0923 17:22:33.274549 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:33.274567 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:33.274755 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:33.275264 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:33.275271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:33.275788 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:33.275946 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:33.275955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:33.277128 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:33.277132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:33.278107 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:33.278119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:33.279244 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:33.279249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:33.279381 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:33.280033 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:33.280052 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:33.283956 (+ 3904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":320,"threads_started":1}
W20260923 17:22:33.284456 5767 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:33.274015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:56227 (local address 127.28.72.254:36239)
0923 17:22:33.274264 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:33.274273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:33.274681 (+ 408us) server_negotiation.cc:407] Connection header received
0923 17:22:33.274898 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:33.274911 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:33.274980 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:33.275111 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:33.276101 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:33.276976 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:33.278448 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:33.278944 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:33.279079 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:33.284184 (+ 5105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:56227: BlockingRecv error: recv got EOF from 127.28.72.194:56227 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 23 17:22:33 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:33 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:33.828259 5772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:33.818487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:53097)
0923 17:22:33.819005 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:33.819022 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:33.819202 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:33.819644 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:33.819648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:33.820051 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:33.820188 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:33.820196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:33.821357 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:33.821361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:33.822294 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:33.822305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:33.823146 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:33.823153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:33.823278 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:33.824025 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:33.824058 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:33.828030 (+ 3972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":291,"threads_started":1}
W20260923 17:22:33.828711 5773 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:33.818555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:53097 (local address 127.28.72.254:36239)
0923 17:22:33.819355 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:33.819359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:33.819376 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:22:33.819427 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:33.819430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:33.819497 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:33.819586 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:33.820398 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:33.821224 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:33.822474 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:33.822928 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:33.823034 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:33.828262 (+ 5228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:53097: BlockingRecv error: recv got EOF from 127.28.72.195:53097 (error 108)
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":577,"threads_started":1}
Sep 23 17:22:33 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:33 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:33.963964 5774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:33.952844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:55671)
0923 17:22:33.953475 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:33.953492 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:33.953605 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:33.953971 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:33.953985 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:33.954413 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:33.954584 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:33.954594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:33.955938 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:33.955943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:33.957063 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:33.957078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:33.958028 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:33.958035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:33.958240 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:33.958997 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:33.959017 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:33.963498 (+ 4481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":414,"threads_started":1}
W20260923 17:22:33.963994 5773 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:33.952934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:55671 (local address 127.28.72.254:36239)
0923 17:22:33.953141 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:33.953150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:33.953588 (+ 438us) server_negotiation.cc:407] Connection header received
0923 17:22:33.953707 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:33.953711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:33.953771 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:33.953857 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:33.954792 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:33.955785 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:33.957237 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:33.957819 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:33.957999 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:33.963770 (+ 5771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:55671: BlockingRecv error: recv got EOF from 127.28.72.193:55671 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 17:22:34 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:34 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:34.294996 5775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:34.285361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:57343)
0923 17:22:34.285927 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:34.285944 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:34.286106 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:34.286486 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:34.286491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:34.286924 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:34.287098 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:34.287134 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:34.288118 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:34.288124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:34.289149 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:34.289161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:34.289953 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:34.289959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:34.290140 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:34.290940 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:34.290961 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:34.294588 (+ 3627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":316,"threads_started":1}
W20260923 17:22:34.295050 5773 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:34.285584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:57343 (local address 127.28.72.254:36239)
0923 17:22:34.285800 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:34.285808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:34.286033 (+ 225us) server_negotiation.cc:407] Connection header received
0923 17:22:34.286243 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:34.286247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:34.286308 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:34.286456 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:34.287267 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:34.287979 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:34.289285 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:34.289799 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:34.289942 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:34.294799 (+ 4857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:57343: BlockingRecv error: recv got EOF from 127.28.72.194:57343 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 23 17:22:34 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:34 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:34.838553 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:34.829325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:54267)
0923 17:22:34.829816 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:34.829838 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:34.829989 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:34.830486 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:34.830489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:34.830877 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:34.831053 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:34.831061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:34.832333 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:34.832337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:34.833289 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:34.833300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:34.833948 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:34.833953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:34.834068 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:34.834728 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:34.834745 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:34.838297 (+ 3552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":420,"threads_started":1}
W20260923 17:22:34.838812 5777 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:34.829594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:54267 (local address 127.28.72.254:36239)
0923 17:22:34.830223 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:34.830228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:34.830242 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:22:34.830284 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:34.830287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:34.830339 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:34.830434 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:34.831248 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:34.832205 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:34.833439 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:34.833815 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:34.833906 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:34.838542 (+ 4636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:54267: BlockingRecv error: recv got EOF from 127.28.72.195:54267 (error 108)
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":328,"threads_started":1}
Sep 23 17:22:34 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:34 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:34.975507 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:34.965166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:57555)
0923 17:22:34.965809 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:34.965827 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:34.966034 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:34.966413 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:34.966418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:34.966919 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:34.967168 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:34.967184 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:34.968531 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:34.968538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:34.969596 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:34.969609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:34.970390 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:34.970395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:34.970529 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:34.971442 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:34.971460 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:34.975263 (+ 3803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":387,"threads_started":1}
W20260923 17:22:34.975735 5777 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:34.965357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:57555 (local address 127.28.72.254:36239)
0923 17:22:34.965596 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:34.965605 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:34.965954 (+ 349us) server_negotiation.cc:407] Connection header received
0923 17:22:34.966185 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:34.966196 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:34.966260 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:34.966383 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:34.967334 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:34.968329 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:34.969746 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:34.970174 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:34.970274 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:34.975467 (+ 5193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:57555: BlockingRecv error: recv got EOF from 127.28.72.193:57555 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 23 17:22:35 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:35 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:35.305934 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:35.296097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:40905)
0923 17:22:35.296720 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:35.296739 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:35.296854 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:35.297194 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:35.297197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:35.297628 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:35.297795 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:35.297804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:35.298937 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:35.298942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:35.299944 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:35.299992 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:35.300849 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:35.300855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:35.300998 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:35.301813 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:35.301840 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:35.305675 (+ 3835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":351,"threads_started":1}
W20260923 17:22:35.306097 5777 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:35.296187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:40905 (local address 127.28.72.254:36239)
0923 17:22:35.296437 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:35.296441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:35.296845 (+ 404us) server_negotiation.cc:407] Connection header received
0923 17:22:35.296961 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:35.296965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:35.297042 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:35.297137 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:35.297963 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:35.298788 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:35.300115 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:35.300674 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:35.300779 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:35.305837 (+ 5058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:40905: BlockingRecv error: recv got EOF from 127.28.72.194:40905 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 17:22:35 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:35 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:35.850847 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:35.839858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:57275)
0923 17:22:35.840481 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:35.840507 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:35.840659 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:35.841139 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:35.841144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:35.841564 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:35.841744 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:35.841753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:35.843195 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:35.843201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:35.844678 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:35.844694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:35.845519 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:35.845525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:35.845704 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:35.846394 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:35.846417 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:35.850467 (+ 4050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":314,"threads_started":1}
W20260923 17:22:35.851059 5781 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:35.839936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:57275 (local address 127.28.72.254:36239)
0923 17:22:35.840795 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:35.840804 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:35.840824 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:22:35.840889 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:35.840893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:35.840970 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:35.841111 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:35.841911 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:35.842993 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:35.844862 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:35.845356 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:35.845468 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:35.850740 (+ 5272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:57275: BlockingRecv error: recv got EOF from 127.28.72.195:57275 (error 108)
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":585,"threads_started":1}
Sep 23 17:22:35 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:35 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:35.986670 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:35.976649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:47571)
0923 17:22:35.977207 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:35.977225 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:35.977348 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:35.977724 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:35.977727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:35.978149 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:35.978309 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:35.978318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:35.979430 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:35.979434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:35.980636 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:35.980650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:35.981535 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:35.981541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:35.981722 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:35.982424 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:35.982443 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:35.986268 (+ 3825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":329,"threads_started":1}
W20260923 17:22:35.986722 5781 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:35.976760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:47571 (local address 127.28.72.254:36239)
0923 17:22:35.976993 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:35.976997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:35.977331 (+ 334us) server_negotiation.cc:407] Connection header received
0923 17:22:35.977503 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:35.977507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:35.977573 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:35.977664 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:35.978461 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:35.979257 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:35.980815 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:35.981377 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:35.981533 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:35.986502 (+ 4969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:47571: BlockingRecv error: recv got EOF from 127.28.72.193:47571 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 23 17:22:36 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:36 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:36.317106 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:36.307044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:44015)
0923 17:22:36.307675 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:36.307701 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:36.307845 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:36.308250 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:36.308255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:36.308857 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:36.309097 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:36.309111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:36.310385 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:36.310390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:36.311546 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:36.311561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:36.312219 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:36.312223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:36.312353 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:36.313018 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:36.313037 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:36.316838 (+ 3801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":359,"threads_started":1}
W20260923 17:22:36.317305 5781 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:36.307232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:44015 (local address 127.28.72.254:36239)
0923 17:22:36.307552 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:36.307561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:36.307794 (+ 233us) server_negotiation.cc:407] Connection header received
0923 17:22:36.307969 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:36.307973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:36.308036 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:36.308166 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:36.309256 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:36.310262 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:36.311681 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:36.312092 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:36.312188 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:36.317069 (+ 4881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:44015: BlockingRecv error: recv got EOF from 127.28.72.194:44015 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 17:22:36 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:36 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:36.862715 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:36.852101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:48175)
0923 17:22:36.852813 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:36.852842 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:36.853038 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:36.853531 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:36.853535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:36.854144 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:36.854376 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:36.854388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:36.855689 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:36.855693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:36.856731 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:36.856745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:36.857628 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:36.857633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:36.857777 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:36.858406 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:36.858489 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:36.862353 (+ 3864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":288,"threads_started":1}
W20260923 17:22:36.862862 5785 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:36.852204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:48175 (local address 127.28.72.254:36239)
0923 17:22:36.852948 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:36.852956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:36.852977 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:22:36.853210 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:36.853214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:36.853315 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:36.853501 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:36.854573 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:36.855536 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:36.856886 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:36.857462 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:36.857583 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:36.862591 (+ 5008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:48175: BlockingRecv error: recv got EOF from 127.28.72.195:48175 (error 108)
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":521,"threads_started":1}
Sep 23 17:22:36 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:36 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:37.004359 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:36.987796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:49825)
0923 17:22:36.988305 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:36.988325 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:36.988471 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:36.988831 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:36.988835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:36.989232 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:36.989451 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:36.989461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:36.990904 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:36.990912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:36.992300 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:36.992314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:36.993069 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:36.993073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:36.993204 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:36.999904 (+ 6700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:36.999928 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:37.004019 (+ 4091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":280,"threads_started":1}
W20260923 17:22:37.004568 5785 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:36.987932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:49825 (local address 127.28.72.254:36239)
0923 17:22:36.988141 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:36.988145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:36.988428 (+ 283us) server_negotiation.cc:407] Connection header received
0923 17:22:36.988584 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:36.988588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:36.988651 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:36.988747 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:36.989637 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:36.990762 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:36.992445 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:36.992912 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:36.993017 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:37.004273 (+ 11256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:49825: BlockingRecv error: recv got EOF from 127.28.72.193:49825 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 23 17:22:37 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:37 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:37.327836 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:37.318312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:53273)
0923 17:22:37.318957 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:37.318975 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:37.319184 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:37.319660 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:37.319666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:37.320102 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:37.320246 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:37.320255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:37.321267 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:37.321273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:37.322218 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:37.322229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:37.322951 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:37.322957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:37.323153 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:37.323737 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:37.323755 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:37.327512 (+ 3757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":263,"threads_started":1}
W20260923 17:22:37.328053 5785 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:37.318506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:53273 (local address 127.28.72.254:36239)
0923 17:22:37.318701 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:37.318707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:37.319086 (+ 379us) server_negotiation.cc:407] Connection header received
0923 17:22:37.319328 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:37.319334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:37.319503 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:37.319617 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:37.320413 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:37.321132 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:37.322369 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:37.322793 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:37.322930 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:37.327733 (+ 4803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:53273: BlockingRecv error: recv got EOF from 127.28.72.194:53273 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 23 17:22:37 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:37 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:37.872313 5788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:37.863722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:47255)
0923 17:22:37.864252 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:37.864268 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:37.864368 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:37.864648 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:37.864651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:37.865051 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:37.865204 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:37.865212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:37.866373 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:37.866377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:37.867351 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:37.867362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:37.868015 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:37.868019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:37.868136 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:37.868758 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:37.868775 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:37.872060 (+ 3285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":290,"threads_started":1}
W20260923 17:22:37.872749 5789 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:37.863798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:47255 (local address 127.28.72.254:36239)
0923 17:22:37.864299 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:37.864304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:37.864321 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:22:37.864459 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:37.864463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:37.864519 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:37.864610 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:37.865397 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:37.866239 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:37.867484 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:37.867893 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:37.868006 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:37.872476 (+ 4470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:47255: BlockingRecv error: recv got EOF from 127.28.72.195:47255 (error 108)
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":313,"threads_started":1}
Sep 23 17:22:38 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:38 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:38.015100 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:38.005354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:41567)
0923 17:22:38.005874 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:38.005890 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:38.006011 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:38.006287 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:38.006290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:38.006689 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:38.006833 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:38.006840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:38.008179 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:38.008183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:38.009295 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:38.009310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:38.010116 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:38.010120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:38.010249 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:38.010967 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:38.010985 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:38.014869 (+ 3884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":286,"threads_started":1}
W20260923 17:22:38.015378 5789 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:38.005458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:41567 (local address 127.28.72.254:36239)
0923 17:22:38.005660 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:38.005665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:38.005990 (+ 325us) server_negotiation.cc:407] Connection header received
0923 17:22:38.006119 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:38.006122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:38.006185 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:38.006274 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:38.006970 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:38.008022 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:38.009479 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:38.009974 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:38.010079 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:38.015072 (+ 4993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:41567: BlockingRecv error: recv got EOF from 127.28.72.193:41567 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 23 17:22:38 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:38 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:38.338961 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:38.329072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:58877)
0923 17:22:38.329590 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:38.329607 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:38.329760 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:38.330140 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:38.330143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:38.330589 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:38.330768 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:38.330777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:38.331913 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:38.331917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:38.332843 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:38.332854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:38.333624 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:38.333628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:38.333751 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:38.334426 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:38.334447 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:38.338699 (+ 4252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":302,"threads_started":1}
W20260923 17:22:38.339229 5789 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:38.329192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:58877 (local address 127.28.72.254:36239)
0923 17:22:38.329360 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:38.329364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:38.329744 (+ 380us) server_negotiation.cc:407] Connection header received
0923 17:22:38.329875 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:38.329879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:38.329964 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:38.330047 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:38.330921 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:38.331755 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:38.332999 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:38.333478 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:38.333583 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:38.338926 (+ 5343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:58877: BlockingRecv error: recv got EOF from 127.28.72.194:58877 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 17:22:38 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:38 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:38.884181 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:38.873704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:45049)
0923 17:22:38.874358 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:38.874479 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:38.874609 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:38.875014 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:38.875017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:38.875535 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:38.875709 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:38.875718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:38.877181 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:38.877185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:38.878168 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:38.878181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:38.879012 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:38.879018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:38.879236 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:38.880040 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:38.880059 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:38.883928 (+ 3869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":296,"threads_started":1}
W20260923 17:22:38.884678 5793 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:38.873832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:45049 (local address 127.28.72.254:36239)
0923 17:22:38.874629 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:38.874636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:38.874693 (+ 57us) server_negotiation.cc:407] Connection header received
0923 17:22:38.874758 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:38.874764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:38.874862 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:38.874974 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:38.875885 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:38.876863 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:38.878335 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:38.878779 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:38.879003 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:38.884177 (+ 5174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:45049: BlockingRecv error: recv got EOF from 127.28.72.195:45049 (error 108)
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":535,"threads_started":1}
Sep 23 17:22:39 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:39 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:22:39.026664 5793 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:39.016496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:56861 (local address 127.28.72.254:36239)
0923 17:22:39.016680 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:39.016685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:39.017215 (+ 530us) server_negotiation.cc:407] Connection header received
0923 17:22:39.017365 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:39.017369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:39.017442 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:39.017591 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:39.018586 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:39.019663 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:39.021149 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:39.021570 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:39.021672 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:39.026452 (+ 4780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:56861: BlockingRecv error: recv got EOF from 127.28.72.193:56861 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:22:39.026681 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:39.016372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:56861)
0923 17:22:39.017107 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:39.017125 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:39.017252 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:39.017619 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:39.017622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:39.018183 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:39.018460 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:39.018473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:39.019840 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:39.019844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:39.020969 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:39.020982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:39.021720 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:39.021725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:39.021840 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:39.022617 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:39.022636 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:39.026232 (+ 3596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":358,"threads_started":1}
Sep 23 17:22:39 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:39 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:39.349778 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:39.340029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:42071)
0923 17:22:39.340693 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:39.340710 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:39.340824 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:39.341164 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:39.341167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:39.341595 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:39.341765 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:39.341774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:39.342890 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:39.342894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:39.343946 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:39.343960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:39.344694 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:39.344698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:39.344815 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:39.345520 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:39.345539 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:39.349423 (+ 3884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":326,"threads_started":1}
W20260923 17:22:39.349889 5793 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:39.340095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:42071 (local address 127.28.72.254:36239)
0923 17:22:39.340321 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:39.340326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:39.340791 (+ 465us) server_negotiation.cc:407] Connection header received
0923 17:22:39.340950 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:39.340954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:39.341029 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:39.341118 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:39.341957 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:39.342728 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:39.344110 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:39.344538 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:39.344642 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:39.349587 (+ 4945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:42071: BlockingRecv error: recv got EOF from 127.28.72.194:42071 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 23 17:22:39 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:39 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:39.896695 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:39.885259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:36767)
0923 17:22:39.885864 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:39.885882 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:39.885992 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:39.886566 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:39.886573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:39.887008 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:39.887246 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:39.887258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:39.888347 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:39.888354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:39.889775 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:39.889814 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:39.890673 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:39.890679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:39.890886 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:39.891591 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:39.891619 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:39.896215 (+ 4596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":267,"threads_started":1}
W20260923 17:22:39.896764 5797 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:39.885352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:36767 (local address 127.28.72.254:36239)
0923 17:22:39.886233 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:39.886237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:39.886253 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:22:39.886311 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:39.886314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:39.886394 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:39.886524 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:39.887403 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:39.888192 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:39.890031 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:39.890503 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:39.890656 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:39.896490 (+ 5834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:36767: BlockingRecv error: recv got EOF from 127.28.72.195:36767 (error 108)
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":593,"threads_started":1}
Sep 23 17:22:40 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:40 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:40.037295 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:40.027812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:56567)
0923 17:22:40.028512 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:40.028529 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:40.028638 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:40.028949 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:40.028952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:40.029342 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:40.029569 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:40.029580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:40.030775 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:40.030778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:40.031790 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:40.031802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:40.032597 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:40.032601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:40.032708 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:40.033438 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:40.033467 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:40.037050 (+ 3583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":369,"threads_started":1}
W20260923 17:22:40.037737 5797 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:40.027935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:56567 (local address 127.28.72.254:36239)
0923 17:22:40.028115 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:40.028119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:40.028620 (+ 501us) server_negotiation.cc:407] Connection header received
0923 17:22:40.028742 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:40.028745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:40.028814 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:40.028900 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:40.029735 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:40.030526 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:40.031938 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:40.032402 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:40.032501 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:40.037344 (+ 4843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:56567: BlockingRecv error: recv got EOF from 127.28.72.193:56567 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 23 17:22:40 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:40 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:40.361232 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:40.350923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:36439)
0923 17:22:40.351479 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:40.351495 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:40.351658 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:40.352009 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:40.352013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:40.352449 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:40.352617 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:40.352626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:40.354012 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:40.354020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:40.355097 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:40.355157 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:40.356056 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:40.356062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:40.356235 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:40.356968 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:40.356987 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:40.360889 (+ 3902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":304,"threads_started":1}
W20260923 17:22:40.361459 5797 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:40.350967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:36439 (local address 127.28.72.254:36239)
0923 17:22:40.351258 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:40.351267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:40.351585 (+ 318us) server_negotiation.cc:407] Connection header received
0923 17:22:40.351783 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:40.351788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:40.351869 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:40.351982 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:40.352763 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:40.353797 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:40.355315 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:40.355766 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:40.356000 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:40.361166 (+ 5166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:36439: BlockingRecv error: recv got EOF from 127.28.72.194:36439 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 23 17:22:40 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:40 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:40.908473 5800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:40.897948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:47727)
0923 17:22:40.898526 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:40.898550 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:40.898657 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:40.899199 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:40.899204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:40.899828 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:40.900018 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:40.900028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:40.901416 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:40.901424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:40.902392 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:40.902406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:40.903460 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:40.903465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:40.903588 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:40.904236 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:40.904254 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:40.908157 (+ 3903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":275,"threads_started":1}
W20260923 17:22:40.908852 5801 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:40.898100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:47727 (local address 127.28.72.254:36239)
0923 17:22:40.898815 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:40.898823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:40.898844 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:22:40.898916 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:40.898919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:40.898980 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:40.899073 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:40.900190 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:40.901167 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:40.902697 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:40.903100 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:40.903252 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:40.908509 (+ 5257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:47727: BlockingRecv error: recv got EOF from 127.28.72.195:47727 (error 108)
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":518,"threads_started":1}
Sep 23 17:22:41 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:41 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:41.049041 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:41.038411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:57755)
0923 17:22:41.038966 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:41.038984 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:41.039103 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:41.039588 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:41.039592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:41.040118 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:41.040365 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:41.040375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:41.041747 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:41.041753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:41.042875 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:41.042890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:41.043681 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:41.043686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:41.043834 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:41.044558 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:41.044578 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:41.048794 (+ 4216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":296,"threads_started":1}
W20260923 17:22:41.049439 5801 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:41.038549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:57755 (local address 127.28.72.254:36239)
0923 17:22:41.038757 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:41.038762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:41.039084 (+ 322us) server_negotiation.cc:407] Connection header received
0923 17:22:41.039232 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:41.039236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:41.039325 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:41.039494 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:41.040579 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:41.041559 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:41.043051 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:41.043506 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:41.043617 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:41.049037 (+ 5420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:57755: BlockingRecv error: recv got EOF from 127.28.72.193:57755 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 23 17:22:41 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:41 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:41.372641 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:41.362532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:44329)
0923 17:22:41.363090 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:41.363154 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:41.363314 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:41.363653 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:41.363657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:41.364090 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:41.364225 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:41.364234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:41.365372 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:41.365378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:41.366471 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:41.366484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:41.367228 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:41.367235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:41.367415 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:41.368010 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:41.368027 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:41.372216 (+ 4189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":295,"threads_started":1}
W20260923 17:22:41.372711 5801 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:41.362584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:44329 (local address 127.28.72.254:36239)
0923 17:22:41.362908 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:41.362915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:41.363240 (+ 325us) server_negotiation.cc:407] Connection header received
0923 17:22:41.363449 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:41.363453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:41.363525 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:41.363628 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:41.364366 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:41.365236 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:41.366613 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:41.367043 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:41.367209 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:41.372429 (+ 5220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:44329: BlockingRecv error: recv got EOF from 127.28.72.194:44329 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
Sep 23 17:22:41 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:41 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:41.919209 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:41.909627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:38401)
0923 17:22:41.910189 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:41.910206 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:41.910313 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:41.910753 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:41.910757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:41.911242 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:41.911452 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:41.911463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:41.912680 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:41.912685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:41.913642 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:41.913653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:41.914493 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:41.914499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:41.914632 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:41.915365 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:41.915383 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:41.918858 (+ 3475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":277,"threads_started":1}
W20260923 17:22:41.919327 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:41.909722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:38401 (local address 127.28.72.254:36239)
0923 17:22:41.910250 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:41.910255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:41.910273 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:22:41.910530 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:41.910535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:41.910611 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:41.910712 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:41.911621 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:41.912526 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:41.913794 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:41.914224 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:41.914464 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:41.919046 (+ 4582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:38401: BlockingRecv error: recv got EOF from 127.28.72.195:38401 (error 108)
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":307,"threads_started":1}
Sep 23 17:22:42 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:42 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:42.063875 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:42.050140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:55517)
0923 17:22:42.050712 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:42.050729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:42.050901 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:42.051317 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:42.051321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:42.051783 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:42.051942 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:42.051952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:42.053030 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:42.053036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:42.054013 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:42.054024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:42.054813 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:42.054819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:42.054992 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:42.059891 (+ 4899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:42.059914 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:42.063528 (+ 3614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":305,"threads_started":1}
W20260923 17:22:42.064010 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:42.050334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:55517 (local address 127.28.72.254:36239)
0923 17:22:42.050589 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:42.050596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:42.050825 (+ 229us) server_negotiation.cc:407] Connection header received
0923 17:22:42.051058 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:42.051063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:42.051155 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:42.051273 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:42.052112 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:42.052860 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:42.054174 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:42.054634 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:42.054774 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:42.063759 (+ 8985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:55517: BlockingRecv error: recv got EOF from 127.28.72.193:55517 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 23 17:22:42 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:42 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:42.384159 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:42.373856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:46631)
0923 17:22:42.374493 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:42.374510 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:42.374683 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:42.375071 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:42.375075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:42.375510 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:42.375683 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:42.375695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:42.376782 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:42.376789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:42.377805 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:42.377817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:42.378531 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:42.378537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:42.378718 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:42.379465 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:42.379488 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:42.383802 (+ 4314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":326,"threads_started":1}
W20260923 17:22:42.384296 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:42.374061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:46631 (local address 127.28.72.254:36239)
0923 17:22:42.374354 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:42.374360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:42.374607 (+ 247us) server_negotiation.cc:407] Connection header received
0923 17:22:42.374815 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:42.374820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:42.374911 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:42.375016 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:42.375843 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:42.376640 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:42.377956 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:42.378365 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:42.378495 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:42.384033 (+ 5538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:46631: BlockingRecv error: recv got EOF from 127.28.72.194:46631 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 23 17:22:42 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:42 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:42.930119 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:42.920346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:56459)
0923 17:22:42.920988 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:42.921011 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:42.921145 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:42.921574 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:42.921579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:42.922142 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:42.922305 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:42.922314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:42.923377 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:42.923381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:42.924372 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:42.924384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:42.925109 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:42.925113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:42.925224 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:42.925800 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:42.925818 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:42.929883 (+ 4065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":369,"threads_started":1}
W20260923 17:22:42.930608 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:42.920456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:56459 (local address 127.28.72.254:36239)
0923 17:22:42.920972 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:42.920979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:42.921101 (+ 122us) server_negotiation.cc:407] Connection header received
0923 17:22:42.921303 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:42.921307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:42.921387 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:42.921508 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:42.922442 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:42.923259 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:42.924518 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:42.924967 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:42.925078 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:42.930129 (+ 5051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:56459: BlockingRecv error: recv got EOF from 127.28.72.195:56459 (error 108)
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":283,"threads_started":1}
Sep 23 17:22:43 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:43 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:43.075424 5810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:43.065204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:49079)
0923 17:22:43.065812 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:43.065829 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:43.065950 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:43.066332 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:43.066335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:43.066735 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:43.066904 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:43.066912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:43.068239 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:43.068244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:43.069404 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:43.069426 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:43.070267 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:43.070274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:43.070462 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:43.071211 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:43.071235 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:43.075083 (+ 3848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":383,"threads_started":1}
W20260923 17:22:43.075471 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:43.065262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:49079 (local address 127.28.72.254:36239)
0923 17:22:43.065588 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:43.065597 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:43.065920 (+ 323us) server_negotiation.cc:407] Connection header received
0923 17:22:43.066095 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:43.066099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:43.066162 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:43.066256 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:43.067103 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:43.068107 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:43.069566 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:43.070110 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:43.070249 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:43.075267 (+ 5018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:49079: BlockingRecv error: recv got EOF from 127.28.72.193:49079 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 23 17:22:43 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:43 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:43.394560 5811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:43.385099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:39605)
0923 17:22:43.385555 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:43.385572 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:43.385679 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:43.386136 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:43.386141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:43.386579 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:43.386767 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:43.386776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:43.387977 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:43.387982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:43.388946 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:43.388960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:43.389671 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:43.389675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:43.389826 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:43.390561 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:43.390580 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:43.394117 (+ 3537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":205,"threads_started":1}
W20260923 17:22:43.394618 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:43.385317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:39605 (local address 127.28.72.254:36239)
0923 17:22:43.385594 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:43.385598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:43.385617 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:22:43.385817 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:43.385822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:43.385911 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:43.386009 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:43.386946 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:43.387859 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:43.389107 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:43.389535 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:43.389638 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:43.394365 (+ 4727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:39605: BlockingRecv error: recv got EOF from 127.28.72.194:39605 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 23 17:22:43 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:43 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:43.941116 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:43.931251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:37067)
0923 17:22:43.931797 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:43.931824 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:43.932026 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:43.932676 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:43.932682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:43.933128 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:43.933357 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:43.933370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:43.934620 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:43.934626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:43.935661 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:43.935675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:43.936452 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:43.936465 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:43.936597 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:43.937218 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:43.937236 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:43.940881 (+ 3645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":328,"threads_started":1}
W20260923 17:22:43.941368 5813 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:43.931335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:37067 (local address 127.28.72.254:36239)
0923 17:22:43.932298 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:43.932303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:43.932317 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:22:43.932395 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:43.932402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:43.932510 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:43.932643 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:43.933587 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:43.934476 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:43.935851 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:43.936261 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:43.936371 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:43.941092 (+ 4721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:37067: BlockingRecv error: recv got EOF from 127.28.72.195:37067 (error 108)
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":533,"threads_started":1}
Sep 23 17:22:44 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:44 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:44.086751 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:44.076400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:37535)
0923 17:22:44.076903 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:44.076920 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:44.077045 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:44.077344 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:44.077348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:44.077789 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:44.077966 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:44.077976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:44.079346 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:44.079351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:44.080351 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:44.080369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:44.081324 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:44.081331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:44.081570 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:44.082216 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:44.082234 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:44.086369 (+ 4135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":280,"threads_started":1}
W20260923 17:22:44.086799 5813 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:44.076524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:37535 (local address 127.28.72.254:36239)
0923 17:22:44.076702 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:44.076706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:44.077004 (+ 298us) server_negotiation.cc:407] Connection header received
0923 17:22:44.077152 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:44.077156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:44.077217 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:44.077301 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:44.078154 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:44.079161 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:44.080604 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:44.081123 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:44.081242 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:44.086583 (+ 5341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:37535: BlockingRecv error: recv got EOF from 127.28.72.193:37535 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 17:22:44 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:44 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:44.405754 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:44.395755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:60459)
0923 17:22:44.396241 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:44.396258 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:44.396425 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:44.396793 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:44.396798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:44.397203 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:44.397451 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:44.397461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:44.398544 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:44.398548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:44.399679 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:44.399692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:44.400787 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:44.400793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:44.400998 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:44.401912 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:44.401935 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:44.405390 (+ 3455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":296,"threads_started":1}
W20260923 17:22:44.405874 5813 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:44.395830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:60459 (local address 127.28.72.254:36239)
0923 17:22:44.396071 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:44.396078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:44.396387 (+ 309us) server_negotiation.cc:407] Connection header received
0923 17:22:44.396583 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:44.396587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:44.396651 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:44.396738 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:44.397629 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:44.398392 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:44.399844 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:44.400634 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:44.400751 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:44.405612 (+ 4861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:60459: BlockingRecv error: recv got EOF from 127.28.72.194:60459 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 23 17:22:44 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:44 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:44.952103 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:44.942030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:47705)
0923 17:22:44.942728 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:44.942745 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:44.942849 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:44.943546 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:44.943549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:44.943905 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:44.944071 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:44.944090 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:44.945233 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:44.945237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:44.946221 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:44.946234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:44.947164 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:44.947169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:44.947348 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:44.947978 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:44.947998 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:44.951804 (+ 3806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":447,"threads_started":1}
W20260923 17:22:44.952545 5817 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:44.942175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:47705 (local address 127.28.72.254:36239)
0923 17:22:44.943162 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:44.943168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:44.943192 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:22:44.943240 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:44.943244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:44.943304 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:44.943438 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:44.944245 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:44.945088 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:44.946384 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:44.946890 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:44.947000 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:44.952145 (+ 5145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:47705: BlockingRecv error: recv got EOF from 127.28.72.195:47705 (error 108)
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":739,"threads_started":1}
Sep 23 17:22:45 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:45 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:45.097364 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.087956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:37259)
0923 17:22:45.088455 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:45.088476 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:45.088623 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:45.089035 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:45.089038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:45.089405 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:45.089569 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.089578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.090915 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.090919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.092045 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.092058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.092781 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.092787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.092972 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.093627 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:45.093645 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:45.097039 (+ 3394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":293,"threads_started":1}
W20260923 17:22:45.097416 5817 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:45.088137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:37259 (local address 127.28.72.254:36239)
0923 17:22:45.088319 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.088324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.088599 (+ 275us) server_negotiation.cc:407] Connection header received
0923 17:22:45.088718 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.088721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.088786 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.088880 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:45.089749 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.090743 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.092201 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.092627 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.092770 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.097204 (+ 4434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:37259: BlockingRecv error: recv got EOF from 127.28.72.193:37259 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 23 17:22:45 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:45 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:45.416626 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.406735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:35645)
0923 17:22:45.407452 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:45.407471 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:45.407588 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:45.407976 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:45.407982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:45.408443 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:45.408600 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.408610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.409798 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.409803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.410799 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.410813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.411585 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.411591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.411726 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.412485 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:45.412505 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:45.416299 (+ 3794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":421,"threads_started":1}
W20260923 17:22:45.416910 5817 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:45.406890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:35645 (local address 127.28.72.254:36239)
0923 17:22:45.407149 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.407156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.407560 (+ 404us) server_negotiation.cc:407] Connection header received
0923 17:22:45.407687 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.407691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.407760 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.407842 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:45.408725 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.409683 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.410968 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.411428 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.411535 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.416594 (+ 5059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:35645: BlockingRecv error: recv got EOF from 127.28.72.194:35645 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 23 17:22:45 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:45 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.195@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:45.963850 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.953207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.195:43157)
0923 17:22:45.953754 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:45.953773 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:45.953881 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:45.954304 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:45.954308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:45.954738 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:45.954902 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.954911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.956135 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.956139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.957103 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.957115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.957996 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.958000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.958128 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.958958 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:45.958977 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:45.963476 (+ 4499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":313,"threads_started":1}
W20260923 17:22:45.964035 5821 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:45.953356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:43157 (local address 127.28.72.254:36239)
0923 17:22:45.953825 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.953829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.953847 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:22:45.954048 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.954051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.954126 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.954224 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:45.955100 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.955975 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.957327 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.957843 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.957954 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.963762 (+ 5808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.195:43157: BlockingRecv error: recv got EOF from 127.28.72.195:43157 (error 108)
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":286,"threads_started":1}
Sep 23 17:22:46 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:46 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.193@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:46.108980 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:46.098850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.193:43917)
0923 17:22:46.099616 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:46.099633 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:46.099836 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:46.100226 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:46.100230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:46.100686 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:46.100948 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:46.100981 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:46.102014 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:46.102019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:46.103005 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:46.103018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:46.103735 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:46.103741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:46.103929 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:46.104556 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:46.104573 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:46.108547 (+ 3974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":443,"threads_started":1}
W20260923 17:22:46.109107 5821 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:46.099036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:43917 (local address 127.28.72.254:36239)
0923 17:22:46.099402 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:46.099409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:46.099782 (+ 373us) server_negotiation.cc:407] Connection header received
0923 17:22:46.099999 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:46.100005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:46.100082 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:46.100196 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:46.101102 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:46.101863 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:46.103187 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:46.103581 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:46.103715 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:46.108844 (+ 5129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.193:43917: BlockingRecv error: recv got EOF from 127.28.72.193:43917 (error 108)
Metrics: {"server-negotiator.queue_time_us":222}
Sep 23 17:22:46 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:22:46 dist-test-slave-j0m4 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.72.194@KRBTEST.COM for oryx/127.28.72.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:22:46.427822 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:46.417945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:36239 (local address 127.28.72.194:43119)
0923 17:22:46.418522 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:46.418540 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:46.418703 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:46.419085 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:46.419090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:46.419568 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:46.419752 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:46.419762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:46.420948 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:46.420954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:46.421989 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:46.422001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:46.422759 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:46.422766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:46.422969 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:46.423641 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:46.423660 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:46.427466 (+ 3806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.72.254:36239: Server oryx/127.28.72.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":300,"threads_started":1}
W20260923 17:22:46.427913 5821 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:46.418051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:43119 (local address 127.28.72.254:36239)
0923 17:22:46.418390 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:46.418402 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:46.418638 (+ 236us) server_negotiation.cc:407] Connection header received
0923 17:22:46.418864 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:46.418874 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:46.418937 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:46.419051 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:46.419944 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:46.420782 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:46.422150 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:46.422594 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:46.422732 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:46.427659 (+ 4927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.72.194:43119: BlockingRecv error: recv got EOF from 127.28.72.194:43119 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 17:22:46.525141 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 5349
I20260923 17:22:46.538726 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:46.526631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.28.72.193:44225)
0923 17:22:46.527458 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:46.527466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:46.527495 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:22:46.527563 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:46.527570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:46.527825 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:46.528020 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:46.529573 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:46.530709 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:46.532053 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:46.532651 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:46.532756 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:46.535436 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:46.535445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:46.535461 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:46.535510 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:46.538182 (+ 2672us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.72.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 17:22:46.538477 (+ 295us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49030: Request ticket server kudu/127.28.72.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":398,"threads_started":1}
W20260923 17:22:46.538873 5825 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49030: Request ticket server kudu/127.28.72.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 17:22:46.539012 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 5349: Not authorized: Client connection negotiation failed: client connection to 127.28.72.193:44225: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.72.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 17:22:46.539271 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 5349
I20260923 17:22:46.556350 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 5500
I20260923 17:22:46.569712 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:46.557548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.28.72.194:40755)
0923 17:22:46.558197 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:46.558206 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:46.558241 (+ 35us) server_negotiation.cc:407] Connection header received
0923 17:22:46.558321 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:46.558331 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:46.558657 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:46.558887 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:46.560251 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:46.561379 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:46.562607 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:46.563316 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:46.563423 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:46.566399 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:46.566408 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:46.566424 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:46.566483 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:46.569161 (+ 2678us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.72.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 17:22:46.569463 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33700: Request ticket server kudu/127.28.72.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":285,"threads_started":1}
W20260923 17:22:46.569839 5826 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33700: Request ticket server kudu/127.28.72.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 17:22:46.569861 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 5500: Not authorized: Client connection negotiation failed: client connection to 127.28.72.194:40755: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.72.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 17:22:46.570070 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 5500
I20260923 17:22:46.586787 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 5633
I20260923 17:22:46.601632 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:46.588210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51286 (local address 127.28.72.195:39859)
0923 17:22:46.588811 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:46.588818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:46.588844 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:22:46.588910 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:46.588918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:46.589188 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:46.589356 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:46.591667 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:46.592885 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:46.594308 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:46.594922 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:46.595103 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:46.598057 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:46.598066 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:46.598082 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:46.598126 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:46.601008 (+ 2882us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.72.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 17:22:46.601354 (+ 346us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51286: Request ticket server kudu/127.28.72.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":295,"threads_started":1}
W20260923 17:22:46.601744 5827 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51286: Request ticket server kudu/127.28.72.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 17:22:46.601835 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 5633: Not authorized: Client connection negotiation failed: client connection to 127.28.72.195:39859: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.72.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 17:22:46.602013 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 5633
I20260923 17:22:46.618877 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 5280
I20260923 17:22:46.620484 5342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:46.736614 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 5280
2026-09-23T17:22:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21636 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T17:22:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:22:46Z Disabled control of system clock
I20260923 17:22:46.818491 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:40745
--webserver_interface=127.28.72.254
--webserver_port=0
--builtin_ntp_servers=127.28.72.212:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 17:22:47.159020 5832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:47.159359 5832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:47.159523 5832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:47.166865 5832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:47.166999 5832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:47.167078 5832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:47.167187 5832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:47.167277 5832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:47.178090 5832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:40745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:40745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:47.179522 5832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:47.181347 5832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:47.194725 5837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:47.194891 5840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:47.194773 5838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:47.195240 5832 server_base.cc:1034] running on GCE node
I20260923 17:22:47.196363 5832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:47.198110 5832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:47.199458 5832 hybrid_clock.cc:648] HybridClock initialized: now 1790184167199385 us; error 60 us; skew 500 ppm
I20260923 17:22:47.199935 5832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:22:47.207963 5832 webserver.cc:460] Webserver started at http://127.28.72.254:44061/ using document root <none> and password file <none>
I20260923 17:22:47.208786 5832 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:47.208937 5832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:47.209285 5832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:47.212411 5832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "472d97216ef44583b7c5c0d108e5da79"
format_stamp: "Formatted at 2026-09-23 17:22:47 on dist-test-slave-j0m4"
server_key: "209a2700264da9a53dbf77c56c5068e7"
server_key_iv: "e7c2adcca80a6e976ad1a18df9ab390b"
server_key_version: "encryptionkey@0"
I20260923 17:22:47.213256 5832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "472d97216ef44583b7c5c0d108e5da79"
format_stamp: "Formatted at 2026-09-23 17:22:47 on dist-test-slave-j0m4"
server_key: "209a2700264da9a53dbf77c56c5068e7"
server_key_iv: "e7c2adcca80a6e976ad1a18df9ab390b"
server_key_version: "encryptionkey@0"
I20260923 17:22:47.218951 5832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 17:22:47.223083 5846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:47.224761 5832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:47.224997 5832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "472d97216ef44583b7c5c0d108e5da79"
format_stamp: "Formatted at 2026-09-23 17:22:47 on dist-test-slave-j0m4"
server_key: "209a2700264da9a53dbf77c56c5068e7"
server_key_iv: "e7c2adcca80a6e976ad1a18df9ab390b"
server_key_version: "encryptionkey@0"
I20260923 17:22:47.225267 5832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:47.243981 5832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:47.245009 5832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:47.245363 5832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:47.265496 5832 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:40745
I20260923 17:22:47.265518 5897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:40745 every 8 connection(s)
I20260923 17:22:47.266997 5832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:47.271059 5898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:47.271970 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 5832
I20260923 17:22:47.272202 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:22:47.272838 28963 external_mini_cluster.cc:1428] Setting key 0ab00d2a0c67838f17955def467a42cd
I20260923 17:22:47.286093 5898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79: Bootstrap starting.
I20260923 17:22:47.288528 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:47.275061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38326 (local address 127.28.72.254:40745)
0923 17:22:47.276300 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:47.276319 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:47.276387 (+ 68us) server_negotiation.cc:407] Connection header received
0923 17:22:47.278055 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:47.278085 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:47.278732 (+ 647us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:47.279238 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:47.279771 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:47.281633 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:47.283026 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:47.284186 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:47.284369 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:47.285540 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:47.285551 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:47.285570 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:47.286317 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:47.286420 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:47.286448 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:47.286569 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:47.286782 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 17:22:47.287150 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":348,"threads_started":1}
I20260923 17:22:47.290138 5898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:47.291316 5898 log.cc:824] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:47.294164 5898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79: No bootstrap required, opened a new log
I20260923 17:22:47.301782 5898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472d97216ef44583b7c5c0d108e5da79" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40745 } }
I20260923 17:22:47.302155 5898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:47.302222 5898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472d97216ef44583b7c5c0d108e5da79, State: Initialized, Role: FOLLOWER
I20260923 17:22:47.302846 5898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472d97216ef44583b7c5c0d108e5da79" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40745 } }
I20260923 17:22:47.303117 5898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:47.303305 5898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:47.303532 5898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:47.305708 5898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472d97216ef44583b7c5c0d108e5da79" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40745 } }
I20260923 17:22:47.306154 5898 leader_election.cc:302] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 472d97216ef44583b7c5c0d108e5da79; no voters:
I20260923 17:22:47.306674 5898 leader_election.cc:288] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:22:47.306840 5903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:47.307651 5903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [term 1 LEADER]: Becoming Leader. State: Replica: 472d97216ef44583b7c5c0d108e5da79, State: Running, Role: LEADER
I20260923 17:22:47.308173 5903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472d97216ef44583b7c5c0d108e5da79" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40745 } }
I20260923 17:22:47.308848 5898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:47.310870 5905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 472d97216ef44583b7c5c0d108e5da79. Latest consensus state: current_term: 1 leader_uuid: "472d97216ef44583b7c5c0d108e5da79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472d97216ef44583b7c5c0d108e5da79" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40745 } } }
I20260923 17:22:47.311290 5905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:47.312227 5904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "472d97216ef44583b7c5c0d108e5da79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472d97216ef44583b7c5c0d108e5da79" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40745 } } }
I20260923 17:22:47.312601 5904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:47.314043 5909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:47.320694 5909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:47.330075 5909 catalog_manager.cc:1347] Generated new cluster ID: 83f9a8357fc8472daa22919743aea7c0
I20260923 17:22:47.330209 5909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:47.354254 5909 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:22:47.355693 5909 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:47.361814 5909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 472d97216ef44583b7c5c0d108e5da79: Generated new TSK 0
I20260923 17:22:47.362967 5909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:22:47.377689 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40745
--builtin_ntp_servers=127.28.72.212:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 17:22:47.724463 5922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:47.724766 5922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:47.724911 5922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:47.732600 5922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:47.732722 5922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:47.732892 5922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:22:47.742313 5922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:47.743793 5922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:47.745678 5922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:47.760329 5928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:47.760361 5930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:47.760438 5927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:47.761158 5922 server_base.cc:1034] running on GCE node
I20260923 17:22:47.761819 5922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:47.762876 5922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:47.764144 5922 hybrid_clock.cc:648] HybridClock initialized: now 1790184167764065 us; error 71 us; skew 500 ppm
I20260923 17:22:47.764570 5922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:22:47.767033 5922 webserver.cc:460] Webserver started at http://127.28.72.193:40003/ using document root <none> and password file <none>
I20260923 17:22:47.767817 5922 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:47.767959 5922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:47.768297 5922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:47.771368 5922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "a3d509354e824575ac13c336ea10442d"
format_stamp: "Formatted at 2026-09-23 17:22:47 on dist-test-slave-j0m4"
server_key: "da5377d90d4ed39f9fe4b032672a6998"
server_key_iv: "598aeb0ca86829d1f9796ae5aa2059da"
server_key_version: "encryptionkey@0"
I20260923 17:22:47.772226 5922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "a3d509354e824575ac13c336ea10442d"
format_stamp: "Formatted at 2026-09-23 17:22:47 on dist-test-slave-j0m4"
server_key: "da5377d90d4ed39f9fe4b032672a6998"
server_key_iv: "598aeb0ca86829d1f9796ae5aa2059da"
server_key_version: "encryptionkey@0"
I20260923 17:22:47.777721 5922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:22:47.781487 5936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:47.783044 5922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:22:47.783339 5922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "a3d509354e824575ac13c336ea10442d"
format_stamp: "Formatted at 2026-09-23 17:22:47 on dist-test-slave-j0m4"
server_key: "da5377d90d4ed39f9fe4b032672a6998"
server_key_iv: "598aeb0ca86829d1f9796ae5aa2059da"
server_key_version: "encryptionkey@0"
I20260923 17:22:47.783584 5922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:47.805836 5922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:47.806905 5922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:47.807317 5922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:47.808329 5922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:47.810148 5922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:47.810279 5922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:47.810453 5922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:47.810559 5922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:47.845721 5922 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:38991
I20260923 17:22:47.845818 6048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:38991 every 8 connection(s)
I20260923 17:22:47.847074 5922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:22:47.855729 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 5922
I20260923 17:22:47.855963 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:22:47.856562 28963 external_mini_cluster.cc:1428] Setting key f0795df32764f9b5b5ce9a184d0043b2
I20260923 17:22:47.859588 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40745
--builtin_ntp_servers=127.28.72.212:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 17:22:47.861754 6051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:47.849794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:53643 (local address 127.28.72.254:40745)
0923 17:22:47.850263 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:47.850269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:47.851599 (+ 1330us) server_negotiation.cc:407] Connection header received
0923 17:22:47.853429 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:47.853434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:47.853511 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:47.853613 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:47.854899 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:47.855811 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:47.857707 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:47.858154 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:47.858288 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:47.860311 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:47.860322 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:47.860329 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:47.860358 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:47.860416 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:47.860436 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:47.860542 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:47.860936 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 17:22:47.861092 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":231,"threads_started":1}
I20260923 17:22:47.862520 6052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:47.850415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40745 (local address 127.28.72.193:53643)
0923 17:22:47.851365 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:47.851408 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:47.853036 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:47.853921 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:47.853933 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:47.854038 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:47.854690 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:47.854710 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:47.855995 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:47.855999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:47.857543 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:47.857558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:47.858530 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:47.858538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:47.858798 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:47.859897 (+ 1099us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:22:47.859903 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:47.859976 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:22:47.860083 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:47.860588 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:47.860601 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:47.860610 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:22:47.860840 (+ 230us) client_negotiation.cc:240] Negotiation successful
0923 17:22:47.861214 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":262,"threads_started":1}
I20260923 17:22:47.864323 6049 heartbeater.cc:347] Connected to a master server at 127.28.72.254:40745
I20260923 17:22:47.864763 6049 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:47.865608 6049 heartbeater.cc:511] Master 127.28.72.254:40745 requested a full tablet report, sending...
I20260923 17:22:47.867969 5863 ts_manager.cc:194] Registered new tserver with Master: a3d509354e824575ac13c336ea10442d (127.28.72.193:38991)
I20260923 17:22:47.869448 5863 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.72.193:53643
W20260923 17:22:48.210752 6053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:48.211051 6053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:48.211208 6053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:48.219014 6053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:48.219187 6053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:48.219412 6053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:22:48.228806 6053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:48.230162 6053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:48.232038 6053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:48.245663 6058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:48.245652 6061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:48.245689 6059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:48.246412 6053 server_base.cc:1034] running on GCE node
I20260923 17:22:48.247014 6053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:48.248063 6053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:48.249293 6053 hybrid_clock.cc:648] HybridClock initialized: now 1790184168249242 us; error 48 us; skew 500 ppm
I20260923 17:22:48.249650 6053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:22:48.251950 6053 webserver.cc:460] Webserver started at http://127.28.72.194:34919/ using document root <none> and password file <none>
I20260923 17:22:48.252699 6053 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:48.252830 6053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:48.253162 6053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:48.256171 6053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "a883ace5a5a9420c926d8ed774f8af53"
format_stamp: "Formatted at 2026-09-23 17:22:48 on dist-test-slave-j0m4"
server_key: "e928b58570623626bbb250d6256e38f4"
server_key_iv: "50151475acb6efa75e463928a270000d"
server_key_version: "encryptionkey@0"
I20260923 17:22:48.256984 6053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "a883ace5a5a9420c926d8ed774f8af53"
format_stamp: "Formatted at 2026-09-23 17:22:48 on dist-test-slave-j0m4"
server_key: "e928b58570623626bbb250d6256e38f4"
server_key_iv: "50151475acb6efa75e463928a270000d"
server_key_version: "encryptionkey@0"
I20260923 17:22:48.262271 6053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:22:48.266350 6067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:48.268146 6053 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:22:48.268347 6053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "a883ace5a5a9420c926d8ed774f8af53"
format_stamp: "Formatted at 2026-09-23 17:22:48 on dist-test-slave-j0m4"
server_key: "e928b58570623626bbb250d6256e38f4"
server_key_iv: "50151475acb6efa75e463928a270000d"
server_key_version: "encryptionkey@0"
I20260923 17:22:48.268632 6053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:48.316277 6053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:48.317265 6053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:48.317538 6053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:48.318496 6053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:48.319986 6053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:48.320075 6053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:48.320149 6053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:48.320209 6053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:48.356312 6053 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:45883
I20260923 17:22:48.356343 6179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:45883 every 8 connection(s)
I20260923 17:22:48.357788 6053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:22:48.367225 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 6053
I20260923 17:22:48.367429 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:22:48.367866 28963 external_mini_cluster.cc:1428] Setting key c3029faf5a481c0c91987afc0f4412de
I20260923 17:22:48.371101 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40745
--builtin_ntp_servers=127.28.72.212:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 17:22:48.372220 6051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:48.360725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:36091 (local address 127.28.72.254:40745)
0923 17:22:48.360951 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:48.360957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:48.362728 (+ 1771us) server_negotiation.cc:407] Connection header received
0923 17:22:48.364630 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:48.364634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:48.364703 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:48.364801 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:48.365995 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:48.366863 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:48.368965 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:48.369554 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:48.369667 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:48.371386 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:48.371394 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:48.371399 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:48.371438 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:48.371488 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:48.371511 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:48.371591 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:48.371882 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 17:22:48.371968 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":85}
I20260923 17:22:48.373775 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:48.361330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40745 (local address 127.28.72.194:36091)
0923 17:22:48.362537 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:48.362582 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:48.364265 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:48.365073 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:48.365082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:48.365182 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:48.365780 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:48.365800 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:48.367051 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:48.367055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:48.368799 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:48.368816 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:48.369801 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:48.369807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:48.370106 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:48.370932 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:22:48.370935 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:48.370993 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:22:48.371108 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:48.371614 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:48.371628 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:48.371640 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 17:22:48.371833 (+ 193us) client_negotiation.cc:240] Negotiation successful
0923 17:22:48.372289 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":299,"threads_started":1}
I20260923 17:22:48.375511 6180 heartbeater.cc:347] Connected to a master server at 127.28.72.254:40745
I20260923 17:22:48.375874 6180 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:48.376690 6180 heartbeater.cc:511] Master 127.28.72.254:40745 requested a full tablet report, sending...
I20260923 17:22:48.378211 5863 ts_manager.cc:194] Registered new tserver with Master: a883ace5a5a9420c926d8ed774f8af53 (127.28.72.194:45883)
I20260923 17:22:48.379572 5863 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.72.194:36091
W20260923 17:22:48.715680 6183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:48.716040 6183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:48.716182 6183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:48.723862 6183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:48.723984 6183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:48.724090 6183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:22:48.733909 6183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:48.735266 6183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:48.737154 6183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:48.751464 6189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:48.751825 6188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:48.751892 6191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:48.752345 6183 server_base.cc:1034] running on GCE node
I20260923 17:22:48.753046 6183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:48.754098 6183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:48.755347 6183 hybrid_clock.cc:648] HybridClock initialized: now 1790184168755294 us; error 44 us; skew 500 ppm
I20260923 17:22:48.755697 6183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:22:48.758183 6183 webserver.cc:460] Webserver started at http://127.28.72.195:34705/ using document root <none> and password file <none>
I20260923 17:22:48.758950 6183 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:48.759087 6183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:48.759454 6183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:48.762435 6183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "1ed18a4a1b904da0a80ed6c260e8d48d"
format_stamp: "Formatted at 2026-09-23 17:22:48 on dist-test-slave-j0m4"
server_key: "2111033b44c81229eb1a1d0d2ecc6d2d"
server_key_iv: "6463490799777ee465d2a5da8faccf7a"
server_key_version: "encryptionkey@0"
I20260923 17:22:48.763404 6183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "1ed18a4a1b904da0a80ed6c260e8d48d"
format_stamp: "Formatted at 2026-09-23 17:22:48 on dist-test-slave-j0m4"
server_key: "2111033b44c81229eb1a1d0d2ecc6d2d"
server_key_iv: "6463490799777ee465d2a5da8faccf7a"
server_key_version: "encryptionkey@0"
I20260923 17:22:48.769241 6183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:22:48.773175 6197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:48.774840 6183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:22:48.775040 6183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "1ed18a4a1b904da0a80ed6c260e8d48d"
format_stamp: "Formatted at 2026-09-23 17:22:48 on dist-test-slave-j0m4"
server_key: "2111033b44c81229eb1a1d0d2ecc6d2d"
server_key_iv: "6463490799777ee465d2a5da8faccf7a"
server_key_version: "encryptionkey@0"
I20260923 17:22:48.775333 6183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:48.797821 6183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:48.798763 6183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:48.799099 6183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:48.800284 6183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:48.801887 6183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:48.801983 6183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:48.802187 6183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:48.802265 6183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:48.843696 6183 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:37677
I20260923 17:22:48.843736 6309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:37677 every 8 connection(s)
I20260923 17:22:48.845191 6183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:22:48.849009 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 6183
I20260923 17:22:48.849211 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:22:48.849761 28963 external_mini_cluster.cc:1428] Setting key 0b3b29116ee23803c130372704e64707
I20260923 17:22:48.871667 6051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:48.848354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:55285 (local address 127.28.72.254:40745)
0923 17:22:48.848579 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:48.848586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:48.860457 (+ 11871us) server_negotiation.cc:407] Connection header received
0923 17:22:48.862492 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:48.862499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:48.862586 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:48.862704 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:48.863991 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:48.864806 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:48.867980 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:48.868510 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:48.868619 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:48.870675 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:48.870680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:48.870683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:48.870703 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:48.870736 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:48.870748 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:48.870822 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:48.871224 (+ 402us) server_negotiation.cc:299] Negotiation successful
0923 17:22:48.871326 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 17:22:48.872941 6049 heartbeater.cc:503] Master 127.28.72.254:40745 was elected leader, sending a full tablet report...
I20260923 17:22:48.873567 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:48.849470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40745 (local address 127.28.72.195:55285)
0923 17:22:48.860252 (+ 10782us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:48.860301 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:48.862065 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:48.862997 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:48.863007 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:48.863160 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:48.863801 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:48.863819 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:48.865010 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:48.865020 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:48.867769 (+ 2749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:48.867801 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:48.868782 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:48.868793 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:48.869236 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:48.870186 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:22:48.870191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:48.870265 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:22:48.870402 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:48.870862 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:48.870876 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:48.870888 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 17:22:48.871145 (+ 257us) client_negotiation.cc:240] Negotiation successful
0923 17:22:48.871697 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10092,"thread_start_us":9787,"threads_started":1}
I20260923 17:22:48.875366 6310 heartbeater.cc:347] Connected to a master server at 127.28.72.254:40745
I20260923 17:22:48.875699 6310 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:48.876529 6310 heartbeater.cc:511] Master 127.28.72.254:40745 requested a full tablet report, sending...
I20260923 17:22:48.878211 5863 ts_manager.cc:194] Registered new tserver with Master: 1ed18a4a1b904da0a80ed6c260e8d48d (127.28.72.195:37677)
I20260923 17:22:48.879477 5863 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.72.195:55285
I20260923 17:22:48.879869 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 17:22:48.889354 6051 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:48.887218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38334 (local address 127.28.72.254:40745)
0923 17:22:48.887520 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:48.887531 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:48.888085 (+ 554us) server_negotiation.cc:407] Connection header received
0923 17:22:48.888302 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:48.888312 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:48.888416 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:48.888554 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:48.889003 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38334: BlockingRecv error: recv got EOF from 127.0.0.1:38334 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:22:48.890163 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 5922
I20260923 17:22:48.898869 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:48.891338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.28.72.193:38991)
0923 17:22:48.892003 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:48.892010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:48.892037 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:22:48.892101 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:48.892107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:48.892324 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:48.892530 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:48.893026 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:48.894106 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:48.895516 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:48.896116 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:48.896260 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:48.897418 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:48.897429 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:48.897444 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:48.898016 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:48.898077 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:48.898103 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:48.898229 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:48.898379 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 17:22:48.898515 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":344,"threads_started":1}
I20260923 17:22:48.900290 6044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:48.984045 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 5922
I20260923 17:22:49.006546 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 6053
I20260923 17:22:49.014979 6321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:49.007661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.28.72.194:45883)
0923 17:22:49.008333 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:49.008340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:49.008368 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:22:49.008429 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:49.008461 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:49.008708 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:49.008877 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:49.009279 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:49.010318 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:49.011644 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:49.012201 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:49.012340 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:49.013422 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:49.013432 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:49.013447 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:49.014064 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:49.014112 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:49.014131 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:49.014259 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:49.014447 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 17:22:49.014575 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":377,"threads_started":1}
I20260923 17:22:49.016237 6175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:49.141520 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 6053
I20260923 17:22:49.158504 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 6183
I20260923 17:22:49.167510 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:49.159737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.28.72.195:37677)
0923 17:22:49.160385 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:49.160400 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:49.160430 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:22:49.160491 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:49.160497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:49.160707 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:49.160950 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:49.161416 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:49.162480 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:49.164006 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:49.164586 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:49.164740 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:49.165960 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:49.165971 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:49.165985 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:49.166595 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:49.166642 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:49.166661 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:49.166794 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:49.166959 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 17:22:49.167110 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":335,"threads_started":1}
I20260923 17:22:49.168875 6305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:49.253659 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 6183
I20260923 17:22:49.269847 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 5832
I20260923 17:22:49.271313 5893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:49.372969 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 5832
2026-09-23T17:22:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2630 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T17:22:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:22:49Z Disabled control of system clock
I20260923 17:22:49.447598 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:38275
--webserver_interface=127.28.72.254
--webserver_port=0
--builtin_ntp_servers=127.28.72.212:35983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:38275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 17:22:49.790714 6329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:49.791018 6329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:49.791184 6329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:49.798779 6329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:49.798915 6329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:49.798954 6329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:49.799041 6329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:49.799173 6329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:49.808861 6329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:35983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:38275
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:38275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:49.810356 6329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:49.812289 6329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:49.825234 6334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:49.825309 6337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:49.825213 6335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:49.826049 6329 server_base.cc:1034] running on GCE node
I20260923 17:22:49.826968 6329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:49.828588 6329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:49.829854 6329 hybrid_clock.cc:648] HybridClock initialized: now 1790184169829806 us; error 61 us; skew 500 ppm
I20260923 17:22:49.830252 6329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:22:49.832783 6329 webserver.cc:460] Webserver started at http://127.28.72.254:34381/ using document root <none> and password file <none>
I20260923 17:22:49.833561 6329 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:49.833704 6329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:49.834054 6329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:49.838712 6329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "1ef719db870343d3b4d1f1a50a645bee"
format_stamp: "Formatted at 2026-09-23 17:22:49 on dist-test-slave-j0m4"
server_key: "957f0fbcc1b32b87c4fbbcecd8dbdfdd"
server_key_iv: "4994162410421dca92f17203f9509edd"
server_key_version: "encryptionkey@0"
I20260923 17:22:49.839622 6329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "1ef719db870343d3b4d1f1a50a645bee"
format_stamp: "Formatted at 2026-09-23 17:22:49 on dist-test-slave-j0m4"
server_key: "957f0fbcc1b32b87c4fbbcecd8dbdfdd"
server_key_iv: "4994162410421dca92f17203f9509edd"
server_key_version: "encryptionkey@0"
I20260923 17:22:49.845120 6329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:22:49.849109 6343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:49.850556 6329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:22:49.850745 6329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "1ef719db870343d3b4d1f1a50a645bee"
format_stamp: "Formatted at 2026-09-23 17:22:49 on dist-test-slave-j0m4"
server_key: "957f0fbcc1b32b87c4fbbcecd8dbdfdd"
server_key_iv: "4994162410421dca92f17203f9509edd"
server_key_version: "encryptionkey@0"
I20260923 17:22:49.850967 6329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:49.857017 6329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:49.857836 6329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:49.858148 6329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:49.878089 6329 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:38275
I20260923 17:22:49.878139 6394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:38275 every 8 connection(s)
I20260923 17:22:49.879667 6329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:49.881994 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 6329
I20260923 17:22:49.882208 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:22:49.883080 28963 external_mini_cluster.cc:1428] Setting key bf552596eb9901adeed196c6f2f1f5f7
I20260923 17:22:49.885017 6395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:49.895259 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:49.885302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.28.72.254:38275)
0923 17:22:49.886563 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:49.886601 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:49.886663 (+ 62us) server_negotiation.cc:407] Connection header received
0923 17:22:49.889559 (+ 2896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:49.889610 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:49.890443 (+ 833us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:49.891012 (+ 569us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:49.892567 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:49.892576 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:49.892628 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:49.892654 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:49.892790 (+ 136us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:49.892820 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:49.893219 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:49.893375 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 17:22:49.893825 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":419,"threads_started":1}
I20260923 17:22:49.901299 6395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee: Bootstrap starting.
I20260923 17:22:49.904651 6395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:49.905798 6395 log.cc:824] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:49.908676 6395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee: No bootstrap required, opened a new log
I20260923 17:22:49.916476 6395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef719db870343d3b4d1f1a50a645bee" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 38275 } }
I20260923 17:22:49.916970 6395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:49.917081 6395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef719db870343d3b4d1f1a50a645bee, State: Initialized, Role: FOLLOWER
I20260923 17:22:49.917668 6395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef719db870343d3b4d1f1a50a645bee" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 38275 } }
I20260923 17:22:49.917922 6395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:49.918077 6395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:49.918253 6395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:49.920519 6395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef719db870343d3b4d1f1a50a645bee" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 38275 } }
I20260923 17:22:49.921072 6395 leader_election.cc:302] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ef719db870343d3b4d1f1a50a645bee; no voters:
I20260923 17:22:49.921723 6395 leader_election.cc:288] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:22:49.921876 6400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:49.923028 6400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [term 1 LEADER]: Becoming Leader. State: Replica: 1ef719db870343d3b4d1f1a50a645bee, State: Running, Role: LEADER
I20260923 17:22:49.923676 6400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef719db870343d3b4d1f1a50a645bee" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 38275 } }
I20260923 17:22:49.924104 6395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:49.926563 6401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ef719db870343d3b4d1f1a50a645bee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef719db870343d3b4d1f1a50a645bee" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 38275 } } }
I20260923 17:22:49.926954 6401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:49.926662 6402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ef719db870343d3b4d1f1a50a645bee. Latest consensus state: current_term: 1 leader_uuid: "1ef719db870343d3b4d1f1a50a645bee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef719db870343d3b4d1f1a50a645bee" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 38275 } } }
I20260923 17:22:49.927099 6402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:49.939774 6406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:49.945262 6406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:49.954699 6406 catalog_manager.cc:1347] Generated new cluster ID: 7b2ecc93c28846cbb1210e8f4d8755ef
I20260923 17:22:49.954847 6406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:49.969512 6406 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:22:49.970044 6406 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:49.977169 6406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ef719db870343d3b4d1f1a50a645bee: Generated new TSK 0
I20260923 17:22:49.978451 6406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:22:49.996439 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:38275
--builtin_ntp_servers=127.28.72.212:35983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 17:22:50.370674 6420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:50.370976 6420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:50.371176 6420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:50.378562 6420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:50.378677 6420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:50.378855 6420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:22:50.388250 6420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:35983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:50.389647 6420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:50.391520 6420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:50.405247 6428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:50.405298 6425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:50.405323 6426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:50.406188 6420 server_base.cc:1034] running on GCE node
I20260923 17:22:50.406870 6420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:50.407900 6420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:50.409143 6420 hybrid_clock.cc:648] HybridClock initialized: now 1790184170409100 us; error 59 us; skew 500 ppm
I20260923 17:22:50.409492 6420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:22:50.412134 6420 webserver.cc:460] Webserver started at http://127.28.72.193:46245/ using document root <none> and password file <none>
I20260923 17:22:50.412914 6420 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:50.413053 6420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:50.413393 6420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:50.416527 6420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "c878616b057947c5880ae0ee18e52af1"
format_stamp: "Formatted at 2026-09-23 17:22:50 on dist-test-slave-j0m4"
server_key: "2f726cb2872cceec1b6ca531b8510ca4"
server_key_iv: "59c6c2ccece91ef044906dd5af1aa232"
server_key_version: "encryptionkey@0"
I20260923 17:22:50.417332 6420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "c878616b057947c5880ae0ee18e52af1"
format_stamp: "Formatted at 2026-09-23 17:22:50 on dist-test-slave-j0m4"
server_key: "2f726cb2872cceec1b6ca531b8510ca4"
server_key_iv: "59c6c2ccece91ef044906dd5af1aa232"
server_key_version: "encryptionkey@0"
I20260923 17:22:50.422899 6420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 17:22:50.426702 6434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:50.428139 6420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:50.428324 6420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "c878616b057947c5880ae0ee18e52af1"
format_stamp: "Formatted at 2026-09-23 17:22:50 on dist-test-slave-j0m4"
server_key: "2f726cb2872cceec1b6ca531b8510ca4"
server_key_iv: "59c6c2ccece91ef044906dd5af1aa232"
server_key_version: "encryptionkey@0"
I20260923 17:22:50.428568 6420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:50.434345 6420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:50.435242 6420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:50.435521 6420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:50.436419 6420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:50.437829 6420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:50.437951 6420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:50.438083 6420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:50.438169 6420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:50.472038 6420 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:42601
I20260923 17:22:50.472095 6546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:42601 every 8 connection(s)
I20260923 17:22:50.473408 6420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:22:50.481433 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 6420
I20260923 17:22:50.481658 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:22:50.482017 6549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:50.476178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:51231 (local address 127.28.72.254:38275)
0923 17:22:50.476629 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:50.476639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:50.477953 (+ 1314us) server_negotiation.cc:407] Connection header received
0923 17:22:50.479645 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:50.479649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:50.479701 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:50.479810 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:50.481206 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:50.481212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:50.481218 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:50.481239 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:50.481292 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:50.481309 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:50.481492 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:50.481637 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 17:22:50.481707 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":195,"threads_started":1}
I20260923 17:22:50.482488 28963 external_mini_cluster.cc:1428] Setting key 05584698ad06e4c631468f1b927b268e
I20260923 17:22:50.482999 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:50.476816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:38275 (local address 127.28.72.193:51231)
0923 17:22:50.477792 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:50.477833 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:50.479395 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:50.480148 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:50.480156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:50.480249 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:50.480882 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:22:50.480885 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:50.480923 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:22:50.481018 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:50.481418 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:50.481425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:50.481430 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:22:50.481691 (+ 261us) client_negotiation.cc:240] Negotiation successful
0923 17:22:50.481946 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":316,"threads_started":1}
I20260923 17:22:50.484529 6547 heartbeater.cc:347] Connected to a master server at 127.28.72.254:38275
I20260923 17:22:50.484839 6547 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:50.485311 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:38275
--builtin_ntp_servers=127.28.72.212:35983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 17:22:50.485464 6547 heartbeater.cc:511] Master 127.28.72.254:38275 requested a full tablet report, sending...
I20260923 17:22:50.487555 6360 ts_manager.cc:194] Registered new tserver with Master: c878616b057947c5880ae0ee18e52af1 (127.28.72.193:42601)
W20260923 17:22:50.821956 6551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:50.822248 6551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:50.822337 6551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:50.829859 6551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:50.829981 6551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:50.830093 6551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:22:50.839931 6551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:35983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:50.841312 6551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:50.843207 6551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:50.857916 6559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:50.857895 6557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:50.858399 6556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:50.858840 6551 server_base.cc:1034] running on GCE node
I20260923 17:22:50.859577 6551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:50.860642 6551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:50.861924 6551 hybrid_clock.cc:648] HybridClock initialized: now 1790184170861863 us; error 63 us; skew 500 ppm
I20260923 17:22:50.862318 6551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:22:50.864833 6551 webserver.cc:460] Webserver started at http://127.28.72.194:34925/ using document root <none> and password file <none>
I20260923 17:22:50.865634 6551 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:50.865772 6551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:50.866108 6551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:50.869107 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "3cab5794fd6447bdb44dcf30db5d23c5"
format_stamp: "Formatted at 2026-09-23 17:22:50 on dist-test-slave-j0m4"
server_key: "ee95545e033f1aabacba8fc427347b4b"
server_key_iv: "4b231e113413e3d62fb8ab98d59e7c61"
server_key_version: "encryptionkey@0"
I20260923 17:22:50.869901 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "3cab5794fd6447bdb44dcf30db5d23c5"
format_stamp: "Formatted at 2026-09-23 17:22:50 on dist-test-slave-j0m4"
server_key: "ee95545e033f1aabacba8fc427347b4b"
server_key_iv: "4b231e113413e3d62fb8ab98d59e7c61"
server_key_version: "encryptionkey@0"
I20260923 17:22:50.875454 6551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:22:50.879227 6565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:50.880759 6551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:50.880965 6551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "3cab5794fd6447bdb44dcf30db5d23c5"
format_stamp: "Formatted at 2026-09-23 17:22:50 on dist-test-slave-j0m4"
server_key: "ee95545e033f1aabacba8fc427347b4b"
server_key_iv: "4b231e113413e3d62fb8ab98d59e7c61"
server_key_version: "encryptionkey@0"
I20260923 17:22:50.881211 6551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:50.887240 6551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:50.888111 6551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:50.888392 6551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:50.889355 6551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:50.890827 6551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:50.890962 6551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:50.891091 6551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:50.891204 6551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260923 17:22:50.925542 6551 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:45873
I20260923 17:22:50.925634 6677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:45873 every 8 connection(s)
I20260923 17:22:50.927086 6551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:22:50.929214 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 6551
I20260923 17:22:50.929417 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:22:50.929901 28963 external_mini_cluster.cc:1428] Setting key c4bf7e74291530818690a5ee0d1e5161
I20260923 17:22:50.933060 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:38275
--builtin_ntp_servers=127.28.72.212:35983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 17:22:50.937773 6549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:50.930223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:60639 (local address 127.28.72.254:38275)
0923 17:22:50.930444 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:50.930450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:50.935186 (+ 4736us) server_negotiation.cc:407] Connection header received
0923 17:22:50.935259 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:50.935267 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:50.935331 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:50.935441 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:50.937093 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:50.937098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:50.937101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:50.937118 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:50.937149 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:50.937161 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:50.937276 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:50.937508 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 17:22:50.937560 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:22:50.939152 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:50.930763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:38275 (local address 127.28.72.194:60639)
0923 17:22:50.931932 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:50.931982 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:50.933939 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:50.935836 (+ 1897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:50.935850 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:50.935984 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:50.936725 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:22:50.936731 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:50.936784 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:22:50.936900 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:50.937262 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:50.937273 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:50.937284 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 17:22:50.937570 (+ 286us) client_negotiation.cc:240] Negotiation successful
0923 17:22:50.937838 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":346,"threads_started":1}
I20260923 17:22:50.940653 6678 heartbeater.cc:347] Connected to a master server at 127.28.72.254:38275
I20260923 17:22:50.941054 6678 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:50.941820 6678 heartbeater.cc:511] Master 127.28.72.254:38275 requested a full tablet report, sending...
I20260923 17:22:50.943547 6360 ts_manager.cc:194] Registered new tserver with Master: 3cab5794fd6447bdb44dcf30db5d23c5 (127.28.72.194:45873)
W20260923 17:22:51.284696 6681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:51.285037 6681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:51.285199 6681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:51.292860 6681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:51.293011 6681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:51.293205 6681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:22:51.305738 6681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:35983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:51.307083 6681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:51.308954 6681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:51.323165 6686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:51.323134 6689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:51.323058 6687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:51.323741 6681 server_base.cc:1034] running on GCE node
I20260923 17:22:51.324541 6681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:51.325706 6681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:51.326977 6681 hybrid_clock.cc:648] HybridClock initialized: now 1790184171326892 us; error 51 us; skew 500 ppm
I20260923 17:22:51.327381 6681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:22:51.330034 6681 webserver.cc:460] Webserver started at http://127.28.72.195:39181/ using document root <none> and password file <none>
I20260923 17:22:51.330857 6681 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:51.331002 6681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:51.331395 6681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:51.334673 6681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "18be46914f0a48bbb80cea3f12af5be7"
format_stamp: "Formatted at 2026-09-23 17:22:51 on dist-test-slave-j0m4"
server_key: "7b83beb62dedf1ab2f7f2f51456a259d"
server_key_iv: "33f3d33be8c321d75e05bc2dedb49bc5"
server_key_version: "encryptionkey@0"
I20260923 17:22:51.335578 6681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "18be46914f0a48bbb80cea3f12af5be7"
format_stamp: "Formatted at 2026-09-23 17:22:51 on dist-test-slave-j0m4"
server_key: "7b83beb62dedf1ab2f7f2f51456a259d"
server_key_iv: "33f3d33be8c321d75e05bc2dedb49bc5"
server_key_version: "encryptionkey@0"
I20260923 17:22:51.341818 6681 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 17:22:51.345950 6695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:51.347731 6681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:51.347944 6681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "18be46914f0a48bbb80cea3f12af5be7"
format_stamp: "Formatted at 2026-09-23 17:22:51 on dist-test-slave-j0m4"
server_key: "7b83beb62dedf1ab2f7f2f51456a259d"
server_key_iv: "33f3d33be8c321d75e05bc2dedb49bc5"
server_key_version: "encryptionkey@0"
I20260923 17:22:51.348198 6681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:51.354971 6681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:51.356043 6681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:51.356338 6681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:51.357347 6681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:51.359046 6681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:51.359236 6681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260923 17:22:51.359390 6681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:51.359480 6681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:51.395247 6681 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:38089
I20260923 17:22:51.395248 6807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:38089 every 8 connection(s)
I20260923 17:22:51.396746 6681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:22:51.401413 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 6681
I20260923 17:22:51.401587 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:22:51.402170 28963 external_mini_cluster.cc:1428] Setting key 51a9949c07c7db810555057b6f400fb7
I20260923 17:22:51.406457 6549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:51.399621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:53901 (local address 127.28.72.254:38275)
0923 17:22:51.399806 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:51.399810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:51.401642 (+ 1832us) server_negotiation.cc:407] Connection header received
0923 17:22:51.403944 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:51.403948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:51.404002 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:51.404121 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:51.405789 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:51.405793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:51.405796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:51.405812 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:51.405843 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:51.405856 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:51.405954 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:51.406184 (+ 230us) server_negotiation.cc:299] Negotiation successful
0923 17:22:51.406237 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:22:51.407612 6810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:51.400158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:38275 (local address 127.28.72.195:53901)
0923 17:22:51.401453 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:51.401504 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:51.403607 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:51.404427 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:51.404440 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:51.404563 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:51.405421 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:22:51.405426 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:51.405491 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:22:51.405615 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:51.405957 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:51.405965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:51.405970 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:22:51.406198 (+ 228us) client_negotiation.cc:240] Negotiation successful
0923 17:22:51.406513 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":450,"threads_started":1}
I20260923 17:22:51.409160 6808 heartbeater.cc:347] Connected to a master server at 127.28.72.254:38275
I20260923 17:22:51.409544 6808 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:51.410144 6808 heartbeater.cc:511] Master 127.28.72.254:38275 requested a full tablet report, sending...
I20260923 17:22:51.411623 6360 ts_manager.cc:194] Registered new tserver with Master: 18be46914f0a48bbb80cea3f12af5be7 (127.28.72.195:38089)
I20260923 17:22:51.418072 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 17:22:51.424660 6549 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:51.422928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.28.72.254:38275)
0923 17:22:51.423104 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:51.423108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:51.423718 (+ 610us) server_negotiation.cc:407] Connection header received
0923 17:22:51.423856 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:51.423861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:51.423939 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:51.424061 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:51.424406 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38544: BlockingRecv error: recv got EOF from 127.0.0.1:38544 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:22:51.425411 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 6420
I20260923 17:22:51.429533 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:51.426385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.28.72.193:42601)
0923 17:22:51.426939 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:51.426946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:51.426971 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:22:51.427029 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:51.427040 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:51.427330 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:51.427459 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:51.428715 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:51.428723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:51.428736 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:51.428761 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:51.428811 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:51.428827 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:51.428943 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:51.429133 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 17:22:51.429218 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":291,"threads_started":1}
I20260923 17:22:51.430899 6542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:51.511409 6547 heartbeater.cc:503] Master 127.28.72.254:38275 was elected leader, sending a full tablet report...
I20260923 17:22:51.513437 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 6420
I20260923 17:22:51.532534 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 6551
I20260923 17:22:51.536702 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:51.533598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.28.72.194:45873)
0923 17:22:51.534173 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:51.534181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:51.534210 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:22:51.534310 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:51.534317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:51.534642 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:51.534853 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:51.535967 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:51.535975 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:51.535987 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:51.536006 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:51.536046 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:51.536061 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:51.536217 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:51.536314 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 17:22:51.536430 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":279,"threads_started":1}
I20260923 17:22:51.538080 6673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:51.619987 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 6551
I20260923 17:22:51.636301 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 6681
I20260923 17:22:51.640832 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:51.637336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43566 (local address 127.28.72.195:38089)
0923 17:22:51.638036 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:51.638043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:51.638076 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:22:51.638140 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:51.638146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:51.638428 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:51.638568 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:51.639795 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:51.639802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:51.639815 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:22:51.639834 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:51.639878 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:22:51.639893 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:51.640028 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:51.640244 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 17:22:51.640413 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":383,"threads_started":1}
I20260923 17:22:51.642369 6803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:51.726109 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 6681
I20260923 17:22:51.742543 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 6329
I20260923 17:22:51.744014 6390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:51.846122 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 6329
2026-09-23T17:22:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2472 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:22:51 dist-test-slave-j0m4 krb5kdc[6826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:22:51 dist-test-slave-j0m4 krb5kdc[6826](info): set up 2 sockets
Sep 23 17:22:51 dist-test-slave-j0m4 krb5kdc[6826](info): commencing operation
krb5kdc: starting...
W20260923 17:22:52.964366 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:22:53 dist-test-slave-j0m4 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:22:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:22:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:22:53.210297 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:35847
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:35847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:53.541615 6839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:53.542023 6839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:53.542182 6839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:53.549588 6839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:53.549769 6839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:53.549885 6839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:53.549993 6839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:53.550069 6839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:53.560225 6839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:35847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:35847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:53.561628 6839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:53.562973 6839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:53.575686 6847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:53.575760 6845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:53.575719 6844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:53.576609 6839 server_base.cc:1034] running on GCE node
I20260923 17:22:53.577579 6839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:53.579041 6839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:53.580250 6839 hybrid_clock.cc:648] HybridClock initialized: now 1790184173580207 us; error 39 us; skew 500 ppm
I20260923 17:22:53.580623 6839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:53 dist-test-slave-j0m4 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184173, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:53.588286 6839 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:22:53.589919 6839 webserver.cc:460] Webserver started at http://127.28.72.254:38685/ using document root <none> and password file <none>
I20260923 17:22:53.590663 6839 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:53.590772 6839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:53.591157 6839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:53.594170 6839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "21f6786b7cc44e79851702331c420e40"
format_stamp: "Formatted at 2026-09-23 17:22:53 on dist-test-slave-j0m4"
server_key: "82e1b5424ecceab762097423423766f6"
server_key_iv: "d47c06b4ba647121cc9b437c9d6042a6"
server_key_version: "encryptionkey@0"
I20260923 17:22:53.594966 6839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "21f6786b7cc44e79851702331c420e40"
format_stamp: "Formatted at 2026-09-23 17:22:53 on dist-test-slave-j0m4"
server_key: "82e1b5424ecceab762097423423766f6"
server_key_iv: "d47c06b4ba647121cc9b437c9d6042a6"
server_key_version: "encryptionkey@0"
I20260923 17:22:53.600224 6839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:22:53.603765 6854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:53.605342 6839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:22:53.605533 6839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "21f6786b7cc44e79851702331c420e40"
format_stamp: "Formatted at 2026-09-23 17:22:53 on dist-test-slave-j0m4"
server_key: "82e1b5424ecceab762097423423766f6"
server_key_iv: "d47c06b4ba647121cc9b437c9d6042a6"
server_key_version: "encryptionkey@0"
I20260923 17:22:53.605741 6839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:53.632254 6839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:53.636937 6839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:53.637236 6839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:53.656353 6839 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:35847
I20260923 17:22:53.656388 6905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:35847 every 8 connection(s)
I20260923 17:22:53.657750 6839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:53.661628 6906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:53.664320 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 6839
I20260923 17:22:53.664551 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:22:53.665196 28963 external_mini_cluster.cc:1428] Setting key a8cb9f6864e6c09d48235e09681d4cdc
I20260923 17:22:53.674834 6906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40: Bootstrap starting.
I20260923 17:22:53.678561 6906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:53.679662 6906 log.cc:824] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:22:53 dist-test-slave-j0m4 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:22:53.682870 6906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40: No bootstrap required, opened a new log
I20260923 17:22:53.691620 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:53.667378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.28.72.254:35847)
0923 17:22:53.668154 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:53.668170 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:53.668220 (+ 50us) server_negotiation.cc:407] Connection header received
0923 17:22:53.670547 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:53.670582 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:53.671189 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:53.671674 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:53.673150 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:53.675147 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:53.676488 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:53.677547 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:53.677737 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:53.682175 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:53.682186 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:53.682208 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:53.682262 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:53.686500 (+ 4238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:53.687425 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:53.687436 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:53.687451 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:53.687666 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:53.688103 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:53.688108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:53.688112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:53.688894 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:53.689216 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:53.689577 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 17:22:53.690090 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":325,"threads_started":1}
I20260923 17:22:53.693719 6906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f6786b7cc44e79851702331c420e40" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35847 } }
I20260923 17:22:53.694104 6906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:53.694214 6906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f6786b7cc44e79851702331c420e40, State: Initialized, Role: FOLLOWER
I20260923 17:22:53.694885 6906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f6786b7cc44e79851702331c420e40" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35847 } }
I20260923 17:22:53.695151 6906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:53.695283 6906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:53.695472 6906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:53.697751 6906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f6786b7cc44e79851702331c420e40" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35847 } }
I20260923 17:22:53.698262 6906 leader_election.cc:302] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21f6786b7cc44e79851702331c420e40; no voters:
I20260923 17:22:53.698890 6906 leader_election.cc:288] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:22:53.699054 6911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:53.699788 6911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [term 1 LEADER]: Becoming Leader. State: Replica: 21f6786b7cc44e79851702331c420e40, State: Running, Role: LEADER
I20260923 17:22:53.700387 6911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f6786b7cc44e79851702331c420e40" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35847 } }
I20260923 17:22:53.701078 6906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:53.703015 6912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21f6786b7cc44e79851702331c420e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f6786b7cc44e79851702331c420e40" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35847 } } }
I20260923 17:22:53.703389 6912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:53.704550 6916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:53.704279 6913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21f6786b7cc44e79851702331c420e40. Latest consensus state: current_term: 1 leader_uuid: "21f6786b7cc44e79851702331c420e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f6786b7cc44e79851702331c420e40" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35847 } } }
I20260923 17:22:53.704909 6913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:53.711345 6916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:53.720335 6916 catalog_manager.cc:1347] Generated new cluster ID: 45cc51956dc94ff8ab9981ce0d865a4c
I20260923 17:22:53.720486 6916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:53.741989 6916 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:22:53.743456 6916 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:53.759333 6916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40: Generated new TSK 0
I20260923 17:22:53.760336 6916 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:22:53.838858 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:35847
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:54.194281 6932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:54.194609 6932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:54.194705 6932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:54.203776 6932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:54.203909 6932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:54.204025 6932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:22:54.214797 6932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:35847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:54.216161 6932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:54.217420 6932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:54.230762 6937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:54.230697 6940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:54.230739 6938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:54.236965 6932 server_base.cc:1034] running on GCE node
I20260923 17:22:54.237756 6932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:54.238828 6932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:54.240082 6932 hybrid_clock.cc:648] HybridClock initialized: now 1790184174239996 us; error 84 us; skew 500 ppm
I20260923 17:22:54.240510 6932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:54 dist-test-slave-j0m4 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184174, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:54.244026 6932 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:22:54.245813 6932 webserver.cc:460] Webserver started at http://127.28.72.193:44355/ using document root <none> and password file <none>
I20260923 17:22:54.246583 6932 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:54.246733 6932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:54.247046 6932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:54.250048 6932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "1ce22b7ad51542acbfb448b470095677"
format_stamp: "Formatted at 2026-09-23 17:22:54 on dist-test-slave-j0m4"
server_key: "6eba73e9f090a8c6674d15dca4e9eda4"
server_key_iv: "95a2acd6c0ac33b6bd324963810dbe12"
server_key_version: "encryptionkey@0"
I20260923 17:22:54.251055 6932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "1ce22b7ad51542acbfb448b470095677"
format_stamp: "Formatted at 2026-09-23 17:22:54 on dist-test-slave-j0m4"
server_key: "6eba73e9f090a8c6674d15dca4e9eda4"
server_key_iv: "95a2acd6c0ac33b6bd324963810dbe12"
server_key_version: "encryptionkey@0"
I20260923 17:22:54.256542 6932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 17:22:54.260421 6947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:54.261940 6932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:22:54.262123 6932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "1ce22b7ad51542acbfb448b470095677"
format_stamp: "Formatted at 2026-09-23 17:22:54 on dist-test-slave-j0m4"
server_key: "6eba73e9f090a8c6674d15dca4e9eda4"
server_key_iv: "95a2acd6c0ac33b6bd324963810dbe12"
server_key_version: "encryptionkey@0"
I20260923 17:22:54.262339 6932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:54.278343 6932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:54.283093 6932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:54.283519 6932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:54.284603 6932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:54.286103 6932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:54.286177 6932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:54.286331 6932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:54.286388 6932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:54.321616 6932 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:44311
I20260923 17:22:54.321662 7059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:44311 every 8 connection(s)
I20260923 17:22:54.322933 6932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:22:54.327389 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 6932
I20260923 17:22:54.327616 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:22:54.328274 28963 external_mini_cluster.cc:1428] Setting key 449059c3daba82ec4d673ff68ec3c78e
Sep 23 17:22:54 dist-test-slave-j0m4 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184174, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:22:54.344200 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:54.325748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:50475 (local address 127.28.72.254:35847)
0923 17:22:54.326199 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:54.326206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:54.327336 (+ 1130us) server_negotiation.cc:407] Connection header received
0923 17:22:54.329356 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:54.329366 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:54.329468 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:54.329588 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:54.331186 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:54.331985 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:54.333624 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:54.334021 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:54.334156 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:54.338009 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:54.338015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:54.338018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:54.338055 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:54.340744 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:54.341603 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:54.341610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:54.341616 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:54.341749 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:54.342223 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:54.342228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:54.342232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:54.342522 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:54.342691 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:54.343561 (+ 870us) server_negotiation.cc:299] Negotiation successful
0923 17:22:54.343817 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":333,"thread_start_us":212,"threads_started":1}
I20260923 17:22:54.345263 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:54.326384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35847 (local address 127.28.72.193:50475)
0923 17:22:54.327163 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:54.327212 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:54.329068 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:54.329856 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:54.329864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:54.330397 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:54.330984 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:54.331002 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:54.332177 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:54.332182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:54.333457 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:54.333469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:54.334260 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:54.334267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:54.334534 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:54.335378 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:54.335404 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:54.337733 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:54.340932 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:54.340943 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:54.340963 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:54.341463 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:54.341875 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:54.341880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:54.341883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:54.342097 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:54.342672 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:54.342681 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:54.343069 (+ 388us) client_negotiation.cc:769] Sending connection context
0923 17:22:54.343462 (+ 393us) client_negotiation.cc:240] Negotiation successful
0923 17:22:54.344057 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":267,"threads_started":1}
I20260923 17:22:54.346916 7060 heartbeater.cc:347] Connected to a master server at 127.28.72.254:35847
I20260923 17:22:54.347328 7060 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:54.348388 7060 heartbeater.cc:511] Master 127.28.72.254:35847 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
I20260923 17:22:54.350982 6871 ts_manager.cc:194] Registered new tserver with Master: 1ce22b7ad51542acbfb448b470095677 (127.28.72.193:44311)
I20260923 17:22:54.352662 6871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:50475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:22:54.383463 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:35847
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:54.717471 7066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:54.717768 7066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:54.717929 7066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:54.725402 7066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:54.725520 7066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:54.725699 7066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:22:54.735692 7066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:35847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:54.737033 7066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:54.738330 7066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:54.752171 7071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:54.752134 7072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:54.752220 7074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:54.753237 7066 server_base.cc:1034] running on GCE node
I20260923 17:22:54.753909 7066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:54.754978 7066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:54.756230 7066 hybrid_clock.cc:648] HybridClock initialized: now 1790184174756186 us; error 60 us; skew 500 ppm
I20260923 17:22:54.756596 7066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:54 dist-test-slave-j0m4 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184174, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:54.760070 7066 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:22:54.761590 7066 webserver.cc:460] Webserver started at http://127.28.72.194:33021/ using document root <none> and password file <none>
I20260923 17:22:54.762271 7066 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:54.762368 7066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:54.762624 7066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:54.765605 7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "a5941dc546ba4e9085788f8850e27d56"
format_stamp: "Formatted at 2026-09-23 17:22:54 on dist-test-slave-j0m4"
server_key: "0062ad9bb3cea895437c1817b77ef4b8"
server_key_iv: "7c2e9774ce1e4fd8ef43b304933e2508"
server_key_version: "encryptionkey@0"
I20260923 17:22:54.766494 7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "a5941dc546ba4e9085788f8850e27d56"
format_stamp: "Formatted at 2026-09-23 17:22:54 on dist-test-slave-j0m4"
server_key: "0062ad9bb3cea895437c1817b77ef4b8"
server_key_iv: "7c2e9774ce1e4fd8ef43b304933e2508"
server_key_version: "encryptionkey@0"
I20260923 17:22:54.772095 7066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260923 17:22:54.775637 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:54.777259 7066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 17:22:54.777478 7066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "a5941dc546ba4e9085788f8850e27d56"
format_stamp: "Formatted at 2026-09-23 17:22:54 on dist-test-slave-j0m4"
server_key: "0062ad9bb3cea895437c1817b77ef4b8"
server_key_iv: "7c2e9774ce1e4fd8ef43b304933e2508"
server_key_version: "encryptionkey@0"
I20260923 17:22:54.777705 7066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:54.801909 7066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:54.806988 7066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:54.807420 7066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:54.808660 7066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:54.810385 7066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:54.810657 7066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:54.810837 7066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:54.810926 7066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:54.846024 7066 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:45915
I20260923 17:22:54.846073 7193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:45915 every 8 connection(s)
I20260923 17:22:54.847693 7066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:22:54.857821 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 7066
I20260923 17:22:54.858052 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:22:54.858500 28963 external_mini_cluster.cc:1428] Setting key 2a4887b199e482bf6956323d9d54de92
Sep 23 17:22:54 dist-test-slave-j0m4 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184174, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:22:54.870527 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:54.850443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:54783 (local address 127.28.72.254:35847)
0923 17:22:54.850913 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:54.850917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:54.851864 (+ 947us) server_negotiation.cc:407] Connection header received
0923 17:22:54.853725 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:54.853731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:54.853823 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:54.853944 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:54.855622 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:54.856329 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:54.857921 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:54.858457 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:54.858606 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:54.862924 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:54.862934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:54.862943 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:54.862994 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:54.866674 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:54.867754 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:54.867763 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:54.867769 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:54.867937 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:54.868374 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:54.868382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:54.868388 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:54.868816 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:54.868998 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:54.869779 (+ 781us) server_negotiation.cc:299] Negotiation successful
0923 17:22:54.870066 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":273,"threads_started":1}
I20260923 17:22:54.871598 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:54.850941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35847 (local address 127.28.72.194:54783)
0923 17:22:54.851703 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:54.851744 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:54.853419 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:54.854203 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:54.854212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:54.854756 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:54.855443 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:54.855461 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:54.856482 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:54.856494 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:54.857764 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:54.857776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:54.858727 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:54.858736 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:54.859021 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:54.859863 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:54.859893 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:54.862648 (+ 2755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:54.866907 (+ 4259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:54.866916 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:54.866932 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:54.867598 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:54.868050 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:54.868054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:54.868057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:54.868252 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:54.868960 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:54.868966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:54.869319 (+ 353us) client_negotiation.cc:769] Sending connection context
0923 17:22:54.869607 (+ 288us) client_negotiation.cc:240] Negotiation successful
0923 17:22:54.870045 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":316,"threads_started":1}
I20260923 17:22:54.873172 7194 heartbeater.cc:347] Connected to a master server at 127.28.72.254:35847
I20260923 17:22:54.873622 7194 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:54.874437 7194 heartbeater.cc:511] Master 127.28.72.254:35847 requested a full tablet report, sending...
I20260923 17:22:54.876077 6871 ts_manager.cc:194] Registered new tserver with Master: a5941dc546ba4e9085788f8850e27d56 (127.28.72.194:45915)
I20260923 17:22:54.876902 6871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:54783
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:22:54.917804 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:35847
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:55.255499 7200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:55.255865 7200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:55.256042 7200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:55.263773 7200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:55.263933 7200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:55.264118 7200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:22:55.274456 7200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:35847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:22:55.276008 7200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:55.277411 7200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:55.291505 7208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:55.291555 7205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:55.291461 7206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:55.292308 7200 server_base.cc:1034] running on GCE node
I20260923 17:22:55.293031 7200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:55.294055 7200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:55.295284 7200 hybrid_clock.cc:648] HybridClock initialized: now 1790184175295209 us; error 66 us; skew 500 ppm
I20260923 17:22:55.295650 7200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:22:55 dist-test-slave-j0m4 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:55.298965 7200 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:22:55.300606 7200 webserver.cc:460] Webserver started at http://127.28.72.195:46271/ using document root <none> and password file <none>
I20260923 17:22:55.301484 7200 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:55.301651 7200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:55.301981 7200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:55.305224 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "1fb5c723261344cf8209704c0cf3493d"
format_stamp: "Formatted at 2026-09-23 17:22:55 on dist-test-slave-j0m4"
server_key: "3d22b6aa72818dabdea2150dcc196728"
server_key_iv: "8ec537bef6ad64acebe79dbe456a9809"
server_key_version: "encryptionkey@0"
I20260923 17:22:55.306053 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "1fb5c723261344cf8209704c0cf3493d"
format_stamp: "Formatted at 2026-09-23 17:22:55 on dist-test-slave-j0m4"
server_key: "3d22b6aa72818dabdea2150dcc196728"
server_key_iv: "8ec537bef6ad64acebe79dbe456a9809"
server_key_version: "encryptionkey@0"
I20260923 17:22:55.312202 7200 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 17:22:55.316105 7215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:55.317598 7200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:22:55.317792 7200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "1fb5c723261344cf8209704c0cf3493d"
format_stamp: "Formatted at 2026-09-23 17:22:55 on dist-test-slave-j0m4"
server_key: "3d22b6aa72818dabdea2150dcc196728"
server_key_iv: "8ec537bef6ad64acebe79dbe456a9809"
server_key_version: "encryptionkey@0"
I20260923 17:22:55.318012 7200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:55.339730 7200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:55.344278 7200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:55.344539 7200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:55.345568 7200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:55.347057 7200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:55.347203 7200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260923 17:22:55.347298 7200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:55.347354 7200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:55.356212 7060 heartbeater.cc:503] Master 127.28.72.254:35847 was elected leader, sending a full tablet report...
I20260923 17:22:55.382738 7200 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:42753
I20260923 17:22:55.382768 7327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:42753 every 8 connection(s)
I20260923 17:22:55.384115 7200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:22:55.391815 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 7200
I20260923 17:22:55.392006 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:22:55.392544 28963 external_mini_cluster.cc:1428] Setting key 17089c8058aba781f4883f27e6334d02
Sep 23 17:22:55 dist-test-slave-j0m4 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:22:55.405319 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.386815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:42219 (local address 127.28.72.254:35847)
0923 17:22:55.387358 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:55.387365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:55.388691 (+ 1326us) server_negotiation.cc:407] Connection header received
0923 17:22:55.390454 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:55.390459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:55.390531 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:55.390636 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:55.392392 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.393474 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.395402 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.395927 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.396051 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.399483 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:55.399488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:55.399492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:55.399526 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:55.402265 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:55.403066 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:55.403070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:55.403073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:55.403308 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:55.403699 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:55.403704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:55.403706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:55.404027 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:55.404147 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:55.404851 (+ 704us) server_negotiation.cc:299] Negotiation successful
0923 17:22:55.405020 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":224,"threads_started":1}
I20260923 17:22:55.406261 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.387639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35847 (local address 127.28.72.195:42219)
0923 17:22:55.388503 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:55.388541 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:55.390176 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:55.390850 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:55.390859 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:55.391507 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:55.392177 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:55.392197 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.393678 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.393682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:55.395269 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:55.395282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.396133 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.396137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:55.396333 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.396977 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:55.396999 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:55.399263 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:55.402386 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:55.402393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:55.402407 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:55.402870 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:55.403411 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:55.403414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:55.403416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:55.403591 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:55.404136 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:55.404142 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:55.404465 (+ 323us) client_negotiation.cc:769] Sending connection context
0923 17:22:55.404694 (+ 229us) client_negotiation.cc:240] Negotiation successful
0923 17:22:55.405091 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":284,"threads_started":1}
I20260923 17:22:55.407876 7328 heartbeater.cc:347] Connected to a master server at 127.28.72.254:35847
I20260923 17:22:55.408214 7328 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:55.409025 7328 heartbeater.cc:511] Master 127.28.72.254:35847 requested a full tablet report, sending...
I20260923 17:22:55.410744 6871 ts_manager.cc:194] Registered new tserver with Master: 1fb5c723261344cf8209704c0cf3493d (127.28.72.195:42753)
I20260923 17:22:55.411599 6871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:42219
I20260923 17:22:55.420683 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:22:55.438941 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.425583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.28.72.254:35847)
0923 17:22:55.425749 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:55.425754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:55.426319 (+ 565us) server_negotiation.cc:407] Connection header received
0923 17:22:55.426469 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:55.426473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:55.426571 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:55.426688 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:55.428191 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.428909 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.430266 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.430704 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.430799 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.433363 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:55.433368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:55.433372 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:55.433408 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:55.436186 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:55.437005 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:55.437010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:55.437012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:55.437132 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:55.437577 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:55.437582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:55.437584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:55.437952 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:55.438151 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:55.438521 (+ 370us) server_negotiation.cc:299] Negotiation successful
0923 17:22:55.438695 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:22:55.443516 6871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:22:55.451254 6871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:22:55.477017 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.465011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:45915 (local address 127.0.0.1:51116)
0923 17:22:55.466015 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:55.466076 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:55.466303 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:55.468686 (+ 2383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:55.468693 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:55.468736 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:55.469386 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:55.469407 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.471317 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.471323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:55.474229 (+ 2906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:55.474240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.475901 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.475907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:55.476091 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.476156 (+ 65us) client_negotiation.cc:769] Sending connection context
0923 17:22:55.476291 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 17:22:55.476537 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":234,"thread_start_us":569,"threads_started":1}
I20260923 17:22:55.477639 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.465603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.28.72.195:42753)
0923 17:22:55.466985 (+ 1382us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:55.466991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:55.467542 (+ 551us) server_negotiation.cc:407] Connection header received
0923 17:22:55.467760 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:55.467772 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:55.468055 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:55.468257 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:55.469697 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.471365 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.474450 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.475918 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.476059 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.476297 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:55.477105 (+ 808us) server_negotiation.cc:299] Negotiation successful
0923 17:22:55.477287 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1218,"thread_start_us":1068,"threads_started":1}
I20260923 17:22:55.477703 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.467915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:44311 (local address 127.0.0.1:37720)
0923 17:22:55.469875 (+ 1960us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:55.469893 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:55.470024 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:55.470736 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:55.470741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:55.470767 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:55.471165 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:55.471177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.472733 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.472738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:55.474676 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:55.474690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.476863 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.476869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:55.477038 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.477069 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:22:55.477150 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:22:55.477274 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1825,"mutex_wait_us":2,"thread_start_us":1671,"threads_started":1}
I20260923 17:22:55.478164 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.465200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.28.72.194:45915)
0923 17:22:55.468097 (+ 2897us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:55.468107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:55.468134 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:22:55.468227 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:55.468234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:55.468473 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:55.468669 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:55.469699 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.471164 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.474434 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.475681 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.475856 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.476114 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:55.477609 (+ 1495us) server_negotiation.cc:299] Negotiation successful
0923 17:22:55.477819 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2730,"thread_start_us":2610,"threads_started":1}
I20260923 17:22:55.478449 7346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.468493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.28.72.193:44311)
0923 17:22:55.469395 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:55.469403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:55.470149 (+ 746us) server_negotiation.cc:407] Connection header received
0923 17:22:55.470253 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:55.470261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:55.470526 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:55.470731 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:55.471377 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.472576 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.474864 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.476341 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.476494 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.476713 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:55.477929 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0923 17:22:55.478105 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":550,"threads_started":1}
I20260923 17:22:55.478482 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.465067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:42753 (local address 127.0.0.1:60182)
0923 17:22:55.467418 (+ 2351us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:55.467435 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:55.467585 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:55.468610 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:55.468621 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:55.468661 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:55.469402 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:55.469418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.471551 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.471556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:55.474226 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:55.474240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.476449 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.476457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:55.476658 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.476705 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 17:22:55.476796 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 17:22:55.476960 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2256,"mutex_wait_us":103,"thread_start_us":2151,"threads_started":1}
I20260923 17:22:55.482484 7263 tablet_service.cc:1467] Processing CreateTablet for tablet 382246a6635e49b5ac0f91c6e3f9ac16 (DEFAULT_TABLE table=test-table [id=5b32717a80364bdab986957971d0bb6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:55.482484 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 382246a6635e49b5ac0f91c6e3f9ac16 (DEFAULT_TABLE table=test-table [id=5b32717a80364bdab986957971d0bb6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:55.484139 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 382246a6635e49b5ac0f91c6e3f9ac16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:55.484138 7263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 382246a6635e49b5ac0f91c6e3f9ac16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:55.486152 6995 tablet_service.cc:1467] Processing CreateTablet for tablet 382246a6635e49b5ac0f91c6e3f9ac16 (DEFAULT_TABLE table=test-table [id=5b32717a80364bdab986957971d0bb6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:55.487855 6995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 382246a6635e49b5ac0f91c6e3f9ac16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:55.500442 7347 tablet_bootstrap.cc:492] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677: Bootstrap starting.
I20260923 17:22:55.503937 7347 tablet_bootstrap.cc:654] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:55.503937 7348 tablet_bootstrap.cc:492] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56: Bootstrap starting.
I20260923 17:22:55.505514 7347 log.cc:824] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:55.507541 7349 tablet_bootstrap.cc:492] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d: Bootstrap starting.
I20260923 17:22:55.507995 7348 tablet_bootstrap.cc:654] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:55.509428 7348 log.cc:824] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:55.509577 7347 tablet_bootstrap.cc:492] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677: No bootstrap required, opened a new log
I20260923 17:22:55.509955 7347 ts_tablet_manager.cc:1397] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 17:22:55.512006 7349 tablet_bootstrap.cc:654] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:55.513365 7349 log.cc:824] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:55.514268 7348 tablet_bootstrap.cc:492] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56: No bootstrap required, opened a new log
I20260923 17:22:55.514595 7348 ts_tablet_manager.cc:1397] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260923 17:22:55.516677 7349 tablet_bootstrap.cc:492] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d: No bootstrap required, opened a new log
I20260923 17:22:55.516959 7349 ts_tablet_manager.cc:1397] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260923 17:22:55.518537 7347 raft_consensus.cc:348] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5941dc546ba4e9085788f8850e27d56" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 45915 } } peers { permanent_uuid: "1fb5c723261344cf8209704c0cf3493d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 42753 } } peers { permanent_uuid: "1ce22b7ad51542acbfb448b470095677" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 44311 } }
I20260923 17:22:55.518929 7347 raft_consensus.cc:374] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:55.519062 7347 raft_consensus.cc:729] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ce22b7ad51542acbfb448b470095677, State: Initialized, Role: FOLLOWER
I20260923 17:22:55.519793 7347 consensus_queue.cc:260] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5941dc546ba4e9085788f8850e27d56" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 45915 } } peers { permanent_uuid: "1fb5c723261344cf8209704c0cf3493d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 42753 } } peers { permanent_uuid: "1ce22b7ad51542acbfb448b470095677" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 44311 } }
I20260923 17:22:55.524550 7347 ts_tablet_manager.cc:1428] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260923 17:22:55.526352 7348 raft_consensus.cc:348] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5941dc546ba4e9085788f8850e27d56" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 45915 } } peers { permanent_uuid: "1fb5c723261344cf8209704c0cf3493d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 42753 } } peers { permanent_uuid: "1ce22b7ad51542acbfb448b470095677" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 44311 } }
I20260923 17:22:55.526888 7348 raft_consensus.cc:374] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:55.527000 7348 raft_consensus.cc:729] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5941dc546ba4e9085788f8850e27d56, State: Initialized, Role: FOLLOWER
I20260923 17:22:55.527112 7349 raft_consensus.cc:348] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5941dc546ba4e9085788f8850e27d56" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 45915 } } peers { permanent_uuid: "1fb5c723261344cf8209704c0cf3493d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 42753 } } peers { permanent_uuid: "1ce22b7ad51542acbfb448b470095677" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 44311 } }
I20260923 17:22:55.527616 7349 raft_consensus.cc:374] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:55.527734 7349 raft_consensus.cc:729] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fb5c723261344cf8209704c0cf3493d, State: Initialized, Role: FOLLOWER
I20260923 17:22:55.527714 7348 consensus_queue.cc:260] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5941dc546ba4e9085788f8850e27d56" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 45915 } } peers { permanent_uuid: "1fb5c723261344cf8209704c0cf3493d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 42753 } } peers { permanent_uuid: "1ce22b7ad51542acbfb448b470095677" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 44311 } }
I20260923 17:22:55.528468 7349 consensus_queue.cc:260] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5941dc546ba4e9085788f8850e27d56" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 45915 } } peers { permanent_uuid: "1fb5c723261344cf8209704c0cf3493d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 42753 } } peers { permanent_uuid: "1ce22b7ad51542acbfb448b470095677" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 44311 } }
I20260923 17:22:55.529624 7348 ts_tablet_manager.cc:1428] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260923 17:22:55.530154 7194 heartbeater.cc:503] Master 127.28.72.254:35847 was elected leader, sending a full tablet report...
I20260923 17:22:55.530640 7328 heartbeater.cc:503] Master 127.28.72.254:35847 was elected leader, sending a full tablet report...
I20260923 17:22:55.532282 7349 ts_tablet_manager.cc:1428] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260923 17:22:55.575517 7061 tablet.cc:2369] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:22:55.599742 7195 tablet.cc:2369] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:22:55.626317 7354 raft_consensus.cc:482] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:55.626652 7354 raft_consensus.cc:504] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5941dc546ba4e9085788f8850e27d56" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 45915 } } peers { permanent_uuid: "1fb5c723261344cf8209704c0cf3493d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 42753 } } peers { permanent_uuid: "1ce22b7ad51542acbfb448b470095677" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 44311 } }
I20260923 17:22:55.628516 7354 leader_election.cc:288] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fb5c723261344cf8209704c0cf3493d (127.28.72.195:42753), 1ce22b7ad51542acbfb448b470095677 (127.28.72.193:44311)
W20260923 17:22:55.635769 7329 tablet.cc:2369] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:22:55.636555 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.629031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:36209 (local address 127.28.72.195:42753)
0923 17:22:55.629211 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:55.629215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:55.629781 (+ 566us) server_negotiation.cc:407] Connection header received
0923 17:22:55.629994 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:55.630000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:55.630088 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:55.630193 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:55.631142 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.632221 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.634223 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.635459 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.635632 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.635718 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:55.636136 (+ 418us) server_negotiation.cc:299] Negotiation successful
0923 17:22:55.636282 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:22:55.636607 7346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.629992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:35399 (local address 127.28.72.193:44311)
0923 17:22:55.630199 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:55.630205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:55.630552 (+ 347us) server_negotiation.cc:407] Connection header received
0923 17:22:55.630690 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:55.630696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:55.630783 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:55.630896 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:55.631670 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.632460 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.634434 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.635334 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.635539 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.635617 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:55.636239 (+ 622us) server_negotiation.cc:299] Negotiation successful
0923 17:22:55.636376 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:22:55.636655 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.628851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:42753 (local address 127.28.72.194:36209)
0923 17:22:55.629644 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:55.629664 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:55.629823 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:55.630481 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:55.630485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:55.630541 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:55.630949 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:55.630962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.632363 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.632368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:55.634044 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:55.634056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.635707 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.635713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:55.635894 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.635962 (+ 68us) client_negotiation.cc:769] Sending connection context
0923 17:22:55.636103 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 17:22:55.636247 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":23,"thread_start_us":375,"threads_started":1}
I20260923 17:22:55.637233 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.629762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:44311 (local address 127.28.72.194:35399)
0923 17:22:55.630448 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:55.630464 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:55.630565 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:55.630956 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:55.630959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:55.630980 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:55.631478 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:55.631492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.632623 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.632627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:55.634238 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:55.634253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.635800 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.635809 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:55.636039 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.636070 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:22:55.636770 (+ 700us) client_negotiation.cc:240] Negotiation successful
0923 17:22:55.636918 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":21,"thread_start_us":392,"threads_started":1}
I20260923 17:22:55.637904 7283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382246a6635e49b5ac0f91c6e3f9ac16" candidate_uuid: "a5941dc546ba4e9085788f8850e27d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fb5c723261344cf8209704c0cf3493d" is_pre_election: true
I20260923 17:22:55.638124 7015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382246a6635e49b5ac0f91c6e3f9ac16" candidate_uuid: "a5941dc546ba4e9085788f8850e27d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ce22b7ad51542acbfb448b470095677" is_pre_election: true
I20260923 17:22:55.638401 7283 raft_consensus.cc:2454] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5941dc546ba4e9085788f8850e27d56 in term 0.
I20260923 17:22:55.638693 7015 raft_consensus.cc:2454] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5941dc546ba4e9085788f8850e27d56 in term 0.
I20260923 17:22:55.639207 7084 leader_election.cc:302] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fb5c723261344cf8209704c0cf3493d, a5941dc546ba4e9085788f8850e27d56; no voters:
I20260923 17:22:55.639678 7354 raft_consensus.cc:2781] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:22:55.639858 7354 raft_consensus.cc:482] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:22:55.639983 7354 raft_consensus.cc:3037] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:55.642215 7354 raft_consensus.cc:504] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5941dc546ba4e9085788f8850e27d56" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 45915 } } peers { permanent_uuid: "1fb5c723261344cf8209704c0cf3493d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 42753 } } peers { permanent_uuid: "1ce22b7ad51542acbfb448b470095677" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 44311 } }
I20260923 17:22:55.642971 7354 leader_election.cc:288] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [CANDIDATE]: Term 1 election: Requested vote from peers 1fb5c723261344cf8209704c0cf3493d (127.28.72.195:42753), 1ce22b7ad51542acbfb448b470095677 (127.28.72.193:44311)
I20260923 17:22:55.643599 7283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382246a6635e49b5ac0f91c6e3f9ac16" candidate_uuid: "a5941dc546ba4e9085788f8850e27d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fb5c723261344cf8209704c0cf3493d"
I20260923 17:22:55.643833 7283 raft_consensus.cc:3037] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:55.643982 7015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382246a6635e49b5ac0f91c6e3f9ac16" candidate_uuid: "a5941dc546ba4e9085788f8850e27d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ce22b7ad51542acbfb448b470095677"
I20260923 17:22:55.644181 7015 raft_consensus.cc:3037] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:55.646071 7283 raft_consensus.cc:2454] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5941dc546ba4e9085788f8850e27d56 in term 1.
I20260923 17:22:55.646358 7015 raft_consensus.cc:2454] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5941dc546ba4e9085788f8850e27d56 in term 1.
I20260923 17:22:55.646500 7084 leader_election.cc:302] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fb5c723261344cf8209704c0cf3493d, a5941dc546ba4e9085788f8850e27d56; no voters:
I20260923 17:22:55.646809 7354 raft_consensus.cc:2781] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:55.647294 7354 raft_consensus.cc:686] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [term 1 LEADER]: Becoming Leader. State: Replica: a5941dc546ba4e9085788f8850e27d56, State: Running, Role: LEADER
I20260923 17:22:55.647876 7354 consensus_queue.cc:237] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5941dc546ba4e9085788f8850e27d56" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 45915 } } peers { permanent_uuid: "1fb5c723261344cf8209704c0cf3493d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 42753 } } peers { permanent_uuid: "1ce22b7ad51542acbfb448b470095677" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 44311 } }
I20260923 17:22:55.652483 6869 catalog_manager.cc:5495] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5941dc546ba4e9085788f8850e27d56 (127.28.72.194). New cstate: current_term: 1 leader_uuid: "a5941dc546ba4e9085788f8850e27d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5941dc546ba4e9085788f8850e27d56" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 45915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fb5c723261344cf8209704c0cf3493d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 42753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ce22b7ad51542acbfb448b470095677" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 44311 } health_report { overall_health: UNKNOWN } } }
I20260923 17:22:55.675660 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.669898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.28.72.194:45915)
0923 17:22:55.670110 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:55.670115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:55.670134 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:22:55.670379 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:55.670385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:55.670480 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:55.670610 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:55.670964 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.671769 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.673203 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.673618 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.673720 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.674317 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:55.674931 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:55.675051 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:55.675232 (+ 181us) server_negotiation.cc:299] Negotiation successful
0923 17:22:55.675301 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 17:22:55.683990 7015 raft_consensus.cc:1261] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677 [term 1 FOLLOWER]: Refusing update from remote peer a5941dc546ba4e9085788f8850e27d56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:55.684075 7283 raft_consensus.cc:1261] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d [term 1 FOLLOWER]: Refusing update from remote peer a5941dc546ba4e9085788f8850e27d56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:55.684979 7358 consensus_queue.cc:1035] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ce22b7ad51542acbfb448b470095677" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 44311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:55.685357 7354 consensus_queue.cc:1035] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fb5c723261344cf8209704c0cf3493d" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 42753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:55.711333 6869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:55.711635 6869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:55.715553 6869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21f6786b7cc44e79851702331c420e40: Sending DeleteTablet for 3 replicas of tablet 382246a6635e49b5ac0f91c6e3f9ac16
I20260923 17:22:55.717041 6995 tablet_service.cc:1514] Processing DeleteTablet for tablet 382246a6635e49b5ac0f91c6e3f9ac16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:55 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:37720
I20260923 17:22:55.717064 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 382246a6635e49b5ac0f91c6e3f9ac16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:55 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:51116
I20260923 17:22:55.717154 7263 tablet_service.cc:1514] Processing DeleteTablet for tablet 382246a6635e49b5ac0f91c6e3f9ac16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:55 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:60182
I20260923 17:22:55.719700 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 6932
I20260923 17:22:55.729475 7371 tablet_replica.cc:330] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677: stopping tablet replica
I20260923 17:22:55.730022 7371 raft_consensus.cc:2229] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:55.730583 7371 raft_consensus.cc:2258] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:55.731642 7373 tablet_replica.cc:330] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d: stopping tablet replica
I20260923 17:22:55.732086 7373 raft_consensus.cc:2229] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:55.732582 7373 raft_consensus.cc:2258] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:55.733403 7372 tablet_replica.cc:330] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56: stopping tablet replica
I20260923 17:22:55.733800 7372 raft_consensus.cc:2229] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:22:55.734509 7372 raft_consensus.cc:2258] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:55.737561 7372 ts_tablet_manager.cc:1905] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:55.737608 7371 ts_tablet_manager.cc:1905] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:55.739006 7373 ts_tablet_manager.cc:1905] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:55.745818 7371 ts_tablet_manager.cc:1918] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:55.746038 7371 log.cc:1196] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/382246a6635e49b5ac0f91c6e3f9ac16
I20260923 17:22:55.746688 7371 ts_tablet_manager.cc:1939] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1ce22b7ad51542acbfb448b470095677: Deleting consensus metadata
Sep 23 17:22:55 dist-test-slave-j0m4 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:22:55.748970 6857 catalog_manager.cc:4841] TS 1ce22b7ad51542acbfb448b470095677 (127.28.72.193:44311): tablet 382246a6635e49b5ac0f91c6e3f9ac16 (table test-table [id=5b32717a80364bdab986957971d0bb6f]) successfully deleted
I20260923 17:22:55.752233 7372 ts_tablet_manager.cc:1918] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:55.752444 7372 log.cc:1196] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/382246a6635e49b5ac0f91c6e3f9ac16
I20260923 17:22:55.753075 7372 ts_tablet_manager.cc:1939] T 382246a6635e49b5ac0f91c6e3f9ac16 P a5941dc546ba4e9085788f8850e27d56: Deleting consensus metadata
I20260923 17:22:55.753712 7373 ts_tablet_manager.cc:1918] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:55.753930 7373 log.cc:1196] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/382246a6635e49b5ac0f91c6e3f9ac16
I20260923 17:22:55.754551 7373 ts_tablet_manager.cc:1939] T 382246a6635e49b5ac0f91c6e3f9ac16 P 1fb5c723261344cf8209704c0cf3493d: Deleting consensus metadata
I20260923 17:22:55.755242 6858 catalog_manager.cc:4841] TS a5941dc546ba4e9085788f8850e27d56 (127.28.72.194:45915): tablet 382246a6635e49b5ac0f91c6e3f9ac16 (table test-table [id=5b32717a80364bdab986957971d0bb6f]) successfully deleted
I20260923 17:22:55.756544 6857 catalog_manager.cc:4841] TS 1fb5c723261344cf8209704c0cf3493d (127.28.72.195:42753): tablet 382246a6635e49b5ac0f91c6e3f9ac16 (table test-table [id=5b32717a80364bdab986957971d0bb6f]) successfully deleted
I20260923 17:22:55.757378 7346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.721869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.28.72.193:44311)
0923 17:22:55.722125 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:55.722136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:55.735735 (+ 13599us) server_negotiation.cc:407] Connection header received
0923 17:22:55.735971 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:55.735977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:55.736086 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:55.736215 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:55.738189 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.739507 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.741620 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.742327 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.742453 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.748962 (+ 6509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:55.748979 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:55.748999 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:55.749056 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:55.753187 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:55.754214 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:55.754225 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:55.754229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:55.754430 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:55.755565 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:55.755572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:55.755576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:55.755982 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:55.756357 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:55.756668 (+ 311us) server_negotiation.cc:299] Negotiation successful
0923 17:22:55.756907 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 17:22:55.758086 7055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:55.899526 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 6932
I20260923 17:22:55.917191 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 7066
Sep 23 17:22:55 dist-test-slave-j0m4 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:22:55.935573 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:55.918051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.28.72.194:45915)
0923 17:22:55.918238 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:55.918243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:55.918270 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:22:55.918518 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:55.918523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:55.918606 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:55.918696 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:55.920316 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.921341 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.923288 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:55.923831 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:55.923959 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:55.929046 (+ 5087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:55.929062 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:55.929078 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:55.929127 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:55.932619 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:55.933394 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:55.933401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:55.933404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:55.933561 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:55.933968 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:55.933972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:55.933974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:55.934341 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:55.934734 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:55.934933 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 17:22:55.935201 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:22:55.936353 7189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:56.054630 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 7066
I20260923 17:22:56.075032 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 7200
Sep 23 17:22:56 dist-test-slave-j0m4 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:22:56.092614 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:56.077138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.28.72.195:42753)
0923 17:22:56.077315 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:56.077322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:56.077348 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:22:56.077401 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:56.077405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:56.077470 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:56.077557 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:56.079098 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:56.079945 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:56.081254 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:56.081761 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:56.081872 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:56.086407 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:56.086432 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:56.086460 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:56.086511 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:56.089626 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:56.090391 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:56.090409 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:56.090411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:56.090585 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:56.090975 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:56.090980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:56.090982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:56.091533 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:56.091774 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:56.092094 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 17:22:56.092273 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:22:56.093328 7323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:56.188504 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 7200
I20260923 17:22:56.206151 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 6839
I20260923 17:22:56.207682 6901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:22:56.322976 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 6839
2026-09-23T17:22:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4478 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:22:56 dist-test-slave-j0m4 krb5kdc[7381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:22:56 dist-test-slave-j0m4 krb5kdc[7381](info): set up 2 sockets
Sep 23 17:22:56 dist-test-slave-j0m4 krb5kdc[7381](info): commencing operation
krb5kdc: starting...
W20260923 17:22:57.452095 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:22:57 dist-test-slave-j0m4 krb5kdc[7381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:22:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:22:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.72.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.72.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/rangeradmin_127.28.72.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/rangeradmin_127.28.72.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.72.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.72.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/rangerlookup_127.28.72.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/rangerlookup_127.28.72.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.72.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.72.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/HTTP_127.28.72.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/HTTP_127.28.72.212@KRBTEST.COM.keytab.
I20260923 17:22:57.798393 28963 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSv3JX5/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/postgres -l logfile start
2026-09-23 17:23:00.669 UTC [7411] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 17:23:00.669 UTC [7411] LOG: listening on IPv4 address "127.28.72.212", port 41251
2026-09-23 17:23:00.674 UTC [7411] LOG: listening on Unix socket "/tmp/.s.PGSQL.41251"
2026-09-23 17:23:00.698 UTC [7414] LOG: database system was shut down at 2026-09-23 17:22:58 UTC
2026-09-23 17:23:00.703 UTC [7411] LOG: database system is ready to accept connections
I20260923 17:23:01.684684 28963 mini_postgres.cc:96] Postgres bound to 41251
2026-09-23 17:23:01.705 UTC [7422] FATAL: database "slave" does not exist
127.28.72.212:41251 - accepting connections
I20260923 17:23:01.706490 28963 mini_ranger.cc:161] Starting Ranger...
I20260923 17:23:01.737161 28963 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 17:23:02.376924 28963 mini_ranger.cc:87] Created ranger Postgres database
I20260923 17:23:02.377105 28963 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-admin
2026-09-23 17:23:02,682 [I] DB FLAVOR :POSTGRES
2026-09-23 17:23:02,683 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:23:02,683 [I] Checking connection..
2026-09-23 17:23:02,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 17:23:02,951 [I] Checking connection passed.
2026-09-23 17:23:02,951 [I] --------- Verifying version history table ---------
2026-09-23 17:23:02,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:23:03,214 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:23:03,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:23:03,474 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:23:03,474 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 17:23:03,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 17:23:03.748 UTC [7486] WARNING: there is no transaction in progress
2026-09-23 17:23:03,754 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 17:23:03,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:23:04,015 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 17:23:04,015 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:23:04,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 17:23:04,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 17:23:04,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j0m4.c.gcp-upstream.internal','N') ;"
2026-09-23 17:23:04,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:23:05,069 [I] Table x_portal_user does not exist in database ranger
2026-09-23 17:23:05,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:23:05,322 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 17:23:05,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:23:05,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:23:05,804 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 17:23:05,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 17:23:06.895 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:06.913 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:06.932 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:06.950 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:06.968 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:06.989 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:07.053 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:07.061 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:07.079 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:07.091 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:07.103 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:07.119 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:07.130 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:07.139 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:07.147 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:07.156 UTC [7594] WARNING: there is no transaction in progress
2026-09-23 17:23:07,912 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 17:23:07,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 17:23:08,144 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 17:23:08,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:23:08,402 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 17:23:08,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:23:08,674 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 17:23:08,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:23:08,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:23:09,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j0m4.c.gcp-upstream.internal';"
2026-09-23 17:23:09,410 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 17:23:09,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 17:23:09,683 [I] DB_PATCHES have already been applied
I20260923 17:23:09.689864 28963 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSv3JX5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 17:23:09.690045 28963 mini_ranger.cc:193] Using host: 127.28.72.212
I20260923 17:23:09.706341 28963 mini_ranger.cc:227] Ranger admin URL: http://127.28.72.212:39467
Sep 23, 2026 5:23:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:23:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:23:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 5:23:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 5:23:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 5:23:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 5:23:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 17:23:10.901911 28963 mini_ranger.cc:160] Time spent starting Ranger: real 9.195s user 0.000s sys 0.045s
Sep 23, 2026 5:23:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 5:23:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 23, 2026 5:23:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 5:23:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 5:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 5:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 5:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 5:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 5:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 5:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 5:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 5:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 5:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 5:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 5:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 5:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 5:23:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 5:23:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 17:23:34.496652 28963 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.72.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.72.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/rangerkms_127.28.72.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/rangerkms_127.28.72.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.72.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.72.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.72.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/HTTP_127.28.72.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.72.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/HTTP_127.28.72.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 17:23:34 dist-test-slave-j0m4 krb5kdc[7381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184214, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 17:23:34.675709 28963 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 17:23:34.698392 28963 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 17:23:35.750562 28963 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 17:23:35.750694 28963 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-kms
I20260923 17:23:35.750900 28963 mini_ranger_kms.cc:227] Using postgres at 127.28.72.212:41251
2026-09-23 17:23:36,192 [I] DB FLAVOR :POSTGRES
2026-09-23 17:23:36,193 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:23:36,193 [I] Checking connection
2026-09-23 17:23:36,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 17:23:36,464 [I] connection success
2026-09-23 17:23:36,464 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 17:23:36,464 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 17:23:36,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 17:23:36,745 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 17:23:36,745 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:23:36,745 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 17:23:36,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:41251/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 17:23:37,021 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 17:23:37.514195 28963 mini_ranger_kms.cc:310] Created kms service
I20260923 17:23:37.724264 28963 mini_ranger_kms.cc:326] Created kudu user
I20260923 17:23:37.773428 28963 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 17:23:37 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184214, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:23:38.209621 28963 mini_ranger_kms.cc:384] Added ranger policy
I20260923 17:23:38.209957 28963 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 17:23:38.210029 28963 mini_ranger_kms.cc:240] Using host: 127.28.72.212
I20260923 17:23:38.232983 28963 mini_ranger_kms.cc:276] Ranger KMS PID: 7775
I20260923 17:23:38.233210 28963 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.72.212:53747
Sep 23, 2026 5:23:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:23:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:23:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 5:23:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 5:23:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 5:23:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 23 17:23:40 dist-test-slave-j0m4 krb5kdc[7381](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184220, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.72.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:40.591310 28963 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.916s user 0.158s sys 0.308s
I20260923 17:23:40.591576 28963 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 17:23:40.591799 28963 mini_ranger_kms.cc:401] 127.28.72.212:53747/kms/v1/keys
Sep 23 17:23:41 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184220, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.72.212@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
Sep 23 17:23:41 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184220, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.72.212@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
Sep 23 17:23:43 dist-test-slave-j0m4 krb5kdc[7381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:23:43.700740 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:43017
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:43017
--ranger_config_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.72.212:53747/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:44.060444 7825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:44.060815 7825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:44.061007 7825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:44.068387 7825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:44.068564 7825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:44.068675 7825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:44.068778 7825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:44.068876 7825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:44.079710 7825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.72.212:53747/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:43017
--ranger_config_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:43017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:23:44.081120 7825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:44.082576 7825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:44.096484 7830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:44.096403 7831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:44.096649 7825 server_base.cc:1034] running on GCE node
W20260923 17:23:44.096503 7833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:44.097894 7825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:44.099543 7825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:44.100807 7825 hybrid_clock.cc:648] HybridClock initialized: now 1790184224100758 us; error 37 us; skew 500 ppm
I20260923 17:23:44.101245 7825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:23:44 dist-test-slave-j0m4 krb5kdc[7381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184224, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:44.105392 7825 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:23:44.107194 7825 webserver.cc:460] Webserver started at http://127.28.72.254:41849/ using document root <none> and password file <none>
I20260923 17:23:44.107901 7825 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:44.108042 7825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:44.108354 7825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:23:44 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184224, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:23:44.168602 7825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "5040feb0e4934af2995b0531ba3fd27d"
format_stamp: "Formatted at 2026-09-23 17:23:44 on dist-test-slave-j0m4"
server_key: "299e4020a797553138abc1bbf197b52d"
server_key_iv: "b7496b519a6afa03d4fb88c6fc526f59"
server_key_version: "kuduclusterkey@0"
I20260923 17:23:44.169533 7825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "5040feb0e4934af2995b0531ba3fd27d"
format_stamp: "Formatted at 2026-09-23 17:23:44 on dist-test-slave-j0m4"
server_key: "299e4020a797553138abc1bbf197b52d"
server_key_iv: "b7496b519a6afa03d4fb88c6fc526f59"
server_key_version: "kuduclusterkey@0"
I20260923 17:23:44.176267 7825 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 17:23:44.198900 7840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:44.200691 7825 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260923 17:23:44.200901 7825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "5040feb0e4934af2995b0531ba3fd27d"
format_stamp: "Formatted at 2026-09-23 17:23:44 on dist-test-slave-j0m4"
server_key: "299e4020a797553138abc1bbf197b52d"
server_key_iv: "b7496b519a6afa03d4fb88c6fc526f59"
server_key_version: "kuduclusterkey@0"
I20260923 17:23:44.201131 7825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:44.222771 7825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:44.227535 7825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:44.227850 7825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:44.249300 7825 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:43017
I20260923 17:23:44.249320 7891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:43017 every 8 connection(s)
I20260923 17:23:44.250867 7825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:44.251590 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 7825
I20260923 17:23:44.251825 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:23:44.252841 28963 external_mini_cluster.cc:1428] Setting key 03b46a0a8dbd7f1b1281eb91dbbd9f07
I20260923 17:23:44.257762 7892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:23:44 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:23:44.270958 7892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d: Bootstrap starting.
I20260923 17:23:44.275384 7892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:44.276587 7892 log.cc:824] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:44.280346 7892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d: No bootstrap required, opened a new log
I20260923 17:23:44.281414 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:44.255948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.28.72.254:43017)
0923 17:23:44.256817 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:44.256833 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:44.256884 (+ 51us) server_negotiation.cc:407] Connection header received
0923 17:23:44.259222 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:44.259253 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:44.259763 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:44.260219 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:44.262135 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:44.264281 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:44.265957 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:44.267037 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:44.267300 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:44.272222 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:44.272234 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:44.272255 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:44.272318 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:44.276699 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:44.277655 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:44.277669 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:44.277678 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:44.277903 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:44.278471 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:44.278478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:44.278485 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:44.279026 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:44.279379 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:44.279840 (+ 461us) server_negotiation.cc:299] Negotiation successful
0923 17:23:44.280294 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":408,"threads_started":1}
I20260923 17:23:44.293040 7892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5040feb0e4934af2995b0531ba3fd27d" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43017 } }
I20260923 17:23:44.293555 7892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:44.293661 7892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5040feb0e4934af2995b0531ba3fd27d, State: Initialized, Role: FOLLOWER
I20260923 17:23:44.294427 7892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5040feb0e4934af2995b0531ba3fd27d" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43017 } }
I20260923 17:23:44.294704 7892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:44.294922 7892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:44.295150 7892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:44.298244 7892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5040feb0e4934af2995b0531ba3fd27d" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43017 } }
I20260923 17:23:44.298854 7892 leader_election.cc:302] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5040feb0e4934af2995b0531ba3fd27d; no voters:
I20260923 17:23:44.299594 7892 leader_election.cc:288] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:44.299706 7897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:44.301216 7897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [term 1 LEADER]: Becoming Leader. State: Replica: 5040feb0e4934af2995b0531ba3fd27d, State: Running, Role: LEADER
I20260923 17:23:44.301899 7892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:44.301816 7897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5040feb0e4934af2995b0531ba3fd27d" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43017 } }
I20260923 17:23:44.305132 7898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5040feb0e4934af2995b0531ba3fd27d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5040feb0e4934af2995b0531ba3fd27d" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43017 } } }
I20260923 17:23:44.305538 7898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:44.307214 7899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5040feb0e4934af2995b0531ba3fd27d. Latest consensus state: current_term: 1 leader_uuid: "5040feb0e4934af2995b0531ba3fd27d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5040feb0e4934af2995b0531ba3fd27d" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43017 } } }
I20260923 17:23:44.307638 7899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:44.309631 7902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:44.316265 7902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:44.337978 7902 catalog_manager.cc:1347] Generated new cluster ID: b7dd9a7c6dfe4ad097391ac760b0d119
I20260923 17:23:44.338119 7902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:44.358382 7902 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:44.360323 7902 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:44.368259 7892 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 17:23:44.401652 7902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d: Generated new TSK 0
I20260923 17:23:44.404280 7902 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:23:45.285758 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.72.212:53747/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:43017
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:23:45 dist-test-slave-j0m4 krb5kdc[7381](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184225, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 17:23:45.728670 7937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:45.729005 7937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:45.729210 7937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:45.738198 7937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:45.738446 7937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:45.738677 7937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:23:45.753371 7937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.72.212:53747/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:23:45.755414 7937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:45.756955 7937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:45.777879 7947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:45.778791 7948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:45.780184 7951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:45.781498 7937 server_base.cc:1034] running on GCE node
I20260923 17:23:45.782322 7937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:45.783656 7937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:45.785003 7937 hybrid_clock.cc:648] HybridClock initialized: now 1790184225784935 us; error 57 us; skew 500 ppm
I20260923 17:23:45.785379 7937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:23:45 dist-test-slave-j0m4 krb5kdc[7381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184225, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:45.796358 7937 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:23:45.798349 7937 webserver.cc:460] Webserver started at http://127.28.72.193:40073/ using document root <none> and password file <none>
I20260923 17:23:45.799305 7937 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:45.799482 7937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:45.799860 7937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:23:45 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184225, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:23:45.824999 7937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "f6a786b358bc487ea9959de22bf75b43"
format_stamp: "Formatted at 2026-09-23 17:23:45 on dist-test-slave-j0m4"
server_key: "a1373883495bbf9eef5751043d9527b0"
server_key_iv: "078ffdbc85904893d1be9ac01d33e66c"
server_key_version: "kuduclusterkey@0"
I20260923 17:23:45.826296 7937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "f6a786b358bc487ea9959de22bf75b43"
format_stamp: "Formatted at 2026-09-23 17:23:45 on dist-test-slave-j0m4"
server_key: "a1373883495bbf9eef5751043d9527b0"
server_key_iv: "078ffdbc85904893d1be9ac01d33e66c"
server_key_version: "kuduclusterkey@0"
I20260923 17:23:45.833542 7937 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s
I20260923 17:23:45.875813 7957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:45.878991 7937 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.011s sys 0.004s
I20260923 17:23:45.879354 7937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "f6a786b358bc487ea9959de22bf75b43"
format_stamp: "Formatted at 2026-09-23 17:23:45 on dist-test-slave-j0m4"
server_key: "a1373883495bbf9eef5751043d9527b0"
server_key_iv: "078ffdbc85904893d1be9ac01d33e66c"
server_key_version: "kuduclusterkey@0"
I20260923 17:23:45.879671 7937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:45.908950 7937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:45.915594 7937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:45.916039 7937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:45.927614 7937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:45.929705 7937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:45.929834 7937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:45.929987 7937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:45.930084 7937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:23:46 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184225, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:23:46.027621 7937 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:33275
I20260923 17:23:46.029505 7937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:23:46.029575 8069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:33275 every 8 connection(s)
I20260923 17:23:46.036042 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 7937
I20260923 17:23:46.036302 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:23:46.037245 28963 external_mini_cluster.cc:1428] Setting key 8b1d12a9637195b4c57d7b2e17bf0d9a
Sep 23 17:23:46 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184225, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:23:46.083503 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:46.040522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:56479 (local address 127.28.72.254:43017)
0923 17:23:46.051521 (+ 10999us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:46.051532 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:46.055066 (+ 3534us) server_negotiation.cc:407] Connection header received
0923 17:23:46.055188 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:46.055197 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:46.055355 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:46.055501 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:46.058542 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:46.060213 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:46.064220 (+ 4007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:46.065261 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:46.065410 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:46.073707 (+ 8297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:46.073719 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:46.073728 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:46.073785 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:46.078340 (+ 4555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:46.079707 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:46.079718 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:46.079725 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:46.079913 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:46.080631 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:46.080638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:46.080642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:46.081114 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:46.081260 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:46.082583 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0923 17:23:46.082901 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10794,"thread_start_us":10672,"threads_started":1}
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
I20260923 17:23:46.086854 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:46.039821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:43017 (local address 127.28.72.193:56479)
0923 17:23:46.052511 (+ 12690us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:46.052575 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:46.054693 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:46.056183 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:46.056204 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:46.057178 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:46.058270 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:46.058303 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:46.060553 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:46.060562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:46.063918 (+ 3356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:46.063947 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:46.065968 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:46.065980 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:46.066584 (+ 604us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:46.069947 (+ 3363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:46.069991 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:46.073330 (+ 3339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:46.078602 (+ 5272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:46.078617 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:46.078645 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:46.079523 (+ 878us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:46.080096 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:46.080104 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:46.080112 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:46.080465 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:46.081399 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:46.081413 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:46.082008 (+ 595us) client_negotiation.cc:769] Sending connection context
0923 17:23:46.084200 (+ 2192us) client_negotiation.cc:240] Negotiation successful
0923 17:23:46.085036 (+ 836us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12338,"thread_start_us":12067,"threads_started":1}
I20260923 17:23:46.089536 8070 heartbeater.cc:347] Connected to a master server at 127.28.72.254:43017
I20260923 17:23:46.089941 8070 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:46.090969 8070 heartbeater.cc:511] Master 127.28.72.254:43017 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
I20260923 17:23:46.094091 7857 ts_manager.cc:194] Registered new tserver with Master: f6a786b358bc487ea9959de22bf75b43 (127.28.72.193:33275)
I20260923 17:23:46.096529 7857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:56479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
W20260923 17:23:46.161836 28963 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.28.72.194: real 0.121s user 0.001s sys 0.019s
I20260923 17:23:46.162827 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.72.212:53747/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:43017
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:23:46 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184225, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
W20260923 17:23:46.680657 8076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:46.681030 8076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:46.681192 8076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:46.689000 8076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:46.689170 8076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:46.689368 8076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:23:46.700762 8076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.72.212:53747/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:23:46.702198 8076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:46.703758 8076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:46.718346 8091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:46.718410 8090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:46.718324 8093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:46.719097 8076 server_base.cc:1034] running on GCE node
I20260923 17:23:46.719756 8076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:46.720712 8076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:46.721972 8076 hybrid_clock.cc:648] HybridClock initialized: now 1790184226721933 us; error 46 us; skew 500 ppm
I20260923 17:23:46.722328 8076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:23:46 dist-test-slave-j0m4 krb5kdc[7381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184226, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:46.726107 8076 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:23:46.727777 8076 webserver.cc:460] Webserver started at http://127.28.72.194:40141/ using document root <none> and password file <none>
I20260923 17:23:46.728469 8076 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:46.728585 8076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:46.728845 8076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:23:46 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184226, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:23:46.743857 8076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "6e3ed8eb33ab4c0b846162b69aeed22f"
format_stamp: "Formatted at 2026-09-23 17:23:46 on dist-test-slave-j0m4"
server_key: "931ea7a7e9a6ce456d8e7e0be97c0361"
server_key_iv: "a5ff1c7a991ec054ad39be0395f39bf2"
server_key_version: "kuduclusterkey@0"
I20260923 17:23:46.744817 8076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "6e3ed8eb33ab4c0b846162b69aeed22f"
format_stamp: "Formatted at 2026-09-23 17:23:46 on dist-test-slave-j0m4"
server_key: "931ea7a7e9a6ce456d8e7e0be97c0361"
server_key_iv: "a5ff1c7a991ec054ad39be0395f39bf2"
server_key_version: "kuduclusterkey@0"
I20260923 17:23:46.750785 8076 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 17:23:46.766748 8100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:46.768816 8076 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:23:46.769054 8076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "6e3ed8eb33ab4c0b846162b69aeed22f"
format_stamp: "Formatted at 2026-09-23 17:23:46 on dist-test-slave-j0m4"
server_key: "931ea7a7e9a6ce456d8e7e0be97c0361"
server_key_iv: "a5ff1c7a991ec054ad39be0395f39bf2"
server_key_version: "kuduclusterkey@0"
I20260923 17:23:46.769349 8076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:46.805119 8076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:46.810786 8076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:46.811218 8076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:46.812569 8076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:46.814530 8076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:46.814631 8076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:46.814765 8076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:46.814831 8076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:46.855556 8076 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:38151
I20260923 17:23:46.855567 8212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:38151 every 8 connection(s)
I20260923 17:23:46.857263 8076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:23:46.859813 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 8076
I20260923 17:23:46.860033 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:23:46.860726 28963 external_mini_cluster.cc:1428] Setting key b9348d8dc38ce46f47a45421c356294b
Sep 23 17:23:46 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184226, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:23:46.889241 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:46.860292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:60007 (local address 127.28.72.254:43017)
0923 17:23:46.862119 (+ 1827us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:46.862125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:46.867548 (+ 5423us) server_negotiation.cc:407] Connection header received
0923 17:23:46.867639 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:46.867647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:46.867749 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:46.867871 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:46.870191 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:46.871667 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:46.874315 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:46.875202 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:46.875374 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:46.880316 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:46.880323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:46.880328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:46.880383 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:46.884605 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:46.885909 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:46.885916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:46.885920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:46.886088 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:46.886825 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:46.886831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:46.886835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:46.887268 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:46.887435 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:46.888578 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0923 17:23:46.888826 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1575,"thread_start_us":1924,"threads_started":1}
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
I20260923 17:23:46.890889 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:46.861010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:43017 (local address 127.28.72.194:60007)
0923 17:23:46.863822 (+ 2812us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:46.863883 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:46.867071 (+ 3188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:46.868201 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:46.868226 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:46.868912 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:46.869926 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:46.869961 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:46.871892 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:46.871898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:46.874070 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:46.874090 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:46.875522 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:46.875531 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:46.875942 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:46.876852 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:46.876894 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:46.880004 (+ 3110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:46.884831 (+ 4827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:46.884849 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:46.884874 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:46.885723 (+ 849us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:46.886258 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:46.886266 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:46.886274 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:46.886638 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:46.887455 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:46.887467 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:46.888045 (+ 578us) client_negotiation.cc:769] Sending connection context
0923 17:23:46.888448 (+ 403us) client_negotiation.cc:240] Negotiation successful
0923 17:23:46.889125 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2462,"thread_start_us":2211,"threads_started":1}
I20260923 17:23:46.892738 8213 heartbeater.cc:347] Connected to a master server at 127.28.72.254:43017
I20260923 17:23:46.893139 8213 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
I20260923 17:23:46.894181 8213 heartbeater.cc:511] Master 127.28.72.254:43017 requested a full tablet report, sending...
I20260923 17:23:46.896365 7857 ts_manager.cc:194] Registered new tserver with Master: 6e3ed8eb33ab4c0b846162b69aeed22f (127.28.72.194:38151)
I20260923 17:23:46.897666 7857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:60007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:23:46.927037 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.72.212:53747/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:43017
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:23:47.101449 8070 heartbeater.cc:503] Master 127.28.72.254:43017 was elected leader, sending a full tablet report...
W20260923 17:23:47.299093 8219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:47.299494 8219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:47.299645 8219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:47.307710 8219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:47.307838 8219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:47.308005 8219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:23:47.318424 8219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.72.212:53747/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:23:47.319892 8219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:47.321357 8219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:47.336144 8225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:47.336144 8224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:47.336236 8227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:47.336920 8219 server_base.cc:1034] running on GCE node
I20260923 17:23:47.337620 8219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:47.338804 8219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:47.340097 8219 hybrid_clock.cc:648] HybridClock initialized: now 1790184227340025 us; error 64 us; skew 500 ppm
I20260923 17:23:47.340574 8219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:23:47 dist-test-slave-j0m4 krb5kdc[7381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184227, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:47.344552 8219 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:23:47.346177 8219 webserver.cc:460] Webserver started at http://127.28.72.195:39177/ using document root <none> and password file <none>
I20260923 17:23:47.346886 8219 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:47.347035 8219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:47.347395 8219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:23:47 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184227, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:23:47.363476 8219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc"
format_stamp: "Formatted at 2026-09-23 17:23:47 on dist-test-slave-j0m4"
server_key: "dfd9e5bab03f5e288cbc7bb5dd8a5045"
server_key_iv: "43b5718f5bdaa5d74a632b06cc9bc292"
server_key_version: "kuduclusterkey@0"
I20260923 17:23:47.364387 8219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc"
format_stamp: "Formatted at 2026-09-23 17:23:47 on dist-test-slave-j0m4"
server_key: "dfd9e5bab03f5e288cbc7bb5dd8a5045"
server_key_iv: "43b5718f5bdaa5d74a632b06cc9bc292"
server_key_version: "kuduclusterkey@0"
I20260923 17:23:47.370179 8219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 17:23:47.386840 8234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:47.388803 8219 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:23:47.389022 8219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc"
format_stamp: "Formatted at 2026-09-23 17:23:47 on dist-test-slave-j0m4"
server_key: "dfd9e5bab03f5e288cbc7bb5dd8a5045"
server_key_iv: "43b5718f5bdaa5d74a632b06cc9bc292"
server_key_version: "kuduclusterkey@0"
I20260923 17:23:47.389297 8219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:47.408851 8219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:47.413648 8219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:47.413954 8219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:47.414917 8219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:47.416447 8219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:47.416532 8219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:47.416621 8219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:47.416666 8219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:47.455174 8219 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:36433
I20260923 17:23:47.455250 8346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:36433 every 8 connection(s)
I20260923 17:23:47.456542 8219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:23:47.462845 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 8219
I20260923 17:23:47.463073 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:23:47.463654 28963 external_mini_cluster.cc:1428] Setting key f5f3cf909a157402a696519ff7a07a6f
Sep 23 17:23:47 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184227, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:23:47.479893 8349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.459276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:43511 (local address 127.28.72.254:43017)
0923 17:23:47.459866 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:47.459871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:47.460723 (+ 852us) server_negotiation.cc:407] Connection header received
0923 17:23:47.462690 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:47.462701 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:47.462787 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:47.462886 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:47.465041 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.466052 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.468153 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.468580 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.468690 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.472513 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:47.472519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:47.472523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:47.472567 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:47.476189 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:47.477108 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:47.477113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:47.477115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:47.477241 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:47.477742 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:47.477746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:47.477748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:47.478060 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:47.478201 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:47.479258 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0923 17:23:47.479502 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":350,"threads_started":1}
I20260923 17:23:47.480911 8350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.459818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:43017 (local address 127.28.72.195:43511)
0923 17:23:47.460556 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:47.460597 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:47.462287 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:47.463244 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:47.463257 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:47.463895 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:47.464809 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:47.464839 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.466240 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.466250 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:47.467961 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:47.467983 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.468794 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.468799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:47.469034 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.469710 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:47.469734 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:47.472196 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:47.476357 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:47.476365 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:47.476382 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:47.476966 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:47.477338 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:47.477341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:47.477343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:47.477604 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:47.478187 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:47.478194 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:47.478673 (+ 479us) client_negotiation.cc:769] Sending connection context
0923 17:23:47.479065 (+ 392us) client_negotiation.cc:240] Negotiation successful
0923 17:23:47.479670 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":291,"threads_started":1}
I20260923 17:23:47.482477 8347 heartbeater.cc:347] Connected to a master server at 127.28.72.254:43017
I20260923 17:23:47.482797 8347 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:47.483644 8347 heartbeater.cc:511] Master 127.28.72.254:43017 requested a full tablet report, sending...
I20260923 17:23:47.485250 7857 ts_manager.cc:194] Registered new tserver with Master: 6e70fe7b9d3f4b87b553c14b0746e9bc (127.28.72.195:36433)
I20260923 17:23:47.486155 7857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:43511
I20260923 17:23:47.492825 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:23:47.512238 8349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.497610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.28.72.254:43017)
0923 17:23:47.497777 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:47.497782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:47.498340 (+ 558us) server_negotiation.cc:407] Connection header received
0923 17:23:47.498506 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:47.498510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:47.498592 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:47.498685 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:47.500191 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.500924 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.502938 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.503475 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.503612 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.506164 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:47.506170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:47.506173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:47.506209 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:47.509167 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:47.510017 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:47.510027 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:47.510030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:47.510166 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:47.510659 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:47.510667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:47.510673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:47.510973 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:47.511274 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:47.511665 (+ 391us) server_negotiation.cc:299] Negotiation successful
0923 17:23:47.511910 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:23:47.517195 7857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:23:47.525018 7857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:23:47.550595 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.539807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36433 (local address 127.0.0.1:50718)
0923 17:23:47.541231 (+ 1424us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:47.541245 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:47.541399 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:47.542536 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:47.542544 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:47.542581 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:47.543506 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:47.543517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.545203 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.545211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:47.547664 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:47.547676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.549595 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.549608 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:47.549813 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.549874 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 17:23:47.549999 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 17:23:47.550196 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1328,"thread_start_us":1608,"threads_started":1}
I20260923 17:23:47.550673 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.539534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.28.72.195:36433)
0923 17:23:47.541057 (+ 1523us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:47.541066 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:47.541409 (+ 343us) server_negotiation.cc:407] Connection header received
0923 17:23:47.541688 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:47.541699 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:47.541961 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:47.542190 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:47.543697 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.545067 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.547861 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.549366 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.549512 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.549766 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:47.550083 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 17:23:47.550275 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1322,"thread_start_us":576,"threads_started":1}
I20260923 17:23:47.551178 8360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.539416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:33275 (local address 127.0.0.1:48644)
0923 17:23:47.541156 (+ 1740us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:47.541203 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:47.541383 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:47.542177 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:47.542186 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:47.542229 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:47.542985 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:47.543003 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.545993 (+ 2990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.546003 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:47.548539 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:47.548552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.550337 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.550344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:47.550535 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.550566 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:23:47.550634 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 17:23:47.550775 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1479,"thread_start_us":1330,"threads_started":1}
I20260923 17:23:47.552191 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.539926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.28.72.193:33275)
0923 17:23:47.540815 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:47.540826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:47.541335 (+ 509us) server_negotiation.cc:407] Connection header received
0923 17:23:47.541626 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:47.541637 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:47.541920 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:47.542891 (+ 971us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:47.543903 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.545864 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.548712 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.550079 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.551019 (+ 940us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.551304 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:47.551669 (+ 365us) server_negotiation.cc:299] Negotiation successful
0923 17:23:47.551859 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":521,"threads_started":1}
I20260923 17:23:47.552572 8364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.541851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:38151 (local address 127.0.0.1:47088)
0923 17:23:47.544266 (+ 2415us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:47.544287 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:47.544406 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:47.545558 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:47.545562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:47.545586 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:47.546082 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:47.546102 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.548122 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.548132 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:47.549730 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:47.549743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.551830 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.551835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:47.552004 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.552033 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 17:23:47.552121 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 17:23:47.552249 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2299,"thread_start_us":2575,"threads_started":1}
I20260923 17:23:47.553388 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.542119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47088 (local address 127.28.72.194:38151)
0923 17:23:47.543822 (+ 1703us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:47.543831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:47.544542 (+ 711us) server_negotiation.cc:407] Connection header received
0923 17:23:47.544652 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:47.544660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:47.545351 (+ 691us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:47.545550 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:47.546322 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.547934 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.549911 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.551370 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.551511 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.551692 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:47.552866 (+ 1174us) server_negotiation.cc:299] Negotiation successful
0923 17:23:47.553054 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1557,"thread_start_us":1448,"threads_started":1}
I20260923 17:23:47.555739 8005 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9bb554fc614f47b12aae97315273de (DEFAULT_TABLE table=test-table [id=1e485c6ef7e349eab5bed3e3f3c40123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:47.555781 8282 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9bb554fc614f47b12aae97315273de (DEFAULT_TABLE table=test-table [id=1e485c6ef7e349eab5bed3e3f3c40123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:47.557299 8005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9bb554fc614f47b12aae97315273de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:47.557310 8282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9bb554fc614f47b12aae97315273de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:47.557686 8148 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9bb554fc614f47b12aae97315273de (DEFAULT_TABLE table=test-table [id=1e485c6ef7e349eab5bed3e3f3c40123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:47.559191 8148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9bb554fc614f47b12aae97315273de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:47.569262 8366 tablet_bootstrap.cc:492] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43: Bootstrap starting.
I20260923 17:23:47.573217 8367 tablet_bootstrap.cc:492] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc: Bootstrap starting.
I20260923 17:23:47.574720 8368 tablet_bootstrap.cc:492] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f: Bootstrap starting.
I20260923 17:23:47.575050 8366 tablet_bootstrap.cc:654] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:47.576653 8366 log.cc:824] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:47.577256 8367 tablet_bootstrap.cc:654] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:47.578472 8368 tablet_bootstrap.cc:654] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:47.578492 8367 log.cc:824] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:47.580238 8368 log.cc:824] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:47.580368 8366 tablet_bootstrap.cc:492] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43: No bootstrap required, opened a new log
I20260923 17:23:47.580775 8366 ts_tablet_manager.cc:1397] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260923 17:23:47.583325 8367 tablet_bootstrap.cc:492] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc: No bootstrap required, opened a new log
I20260923 17:23:47.583638 8367 ts_tablet_manager.cc:1397] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260923 17:23:47.585961 8368 tablet_bootstrap.cc:492] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f: No bootstrap required, opened a new log
I20260923 17:23:47.586277 8368 ts_tablet_manager.cc:1397] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260923 17:23:47.589236 8366 raft_consensus.cc:348] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36433 } } peers { permanent_uuid: "f6a786b358bc487ea9959de22bf75b43" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 33275 } } peers { permanent_uuid: "6e3ed8eb33ab4c0b846162b69aeed22f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38151 } }
I20260923 17:23:47.589670 8366 raft_consensus.cc:374] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:47.589835 8366 raft_consensus.cc:729] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a786b358bc487ea9959de22bf75b43, State: Initialized, Role: FOLLOWER
I20260923 17:23:47.590555 8366 consensus_queue.cc:260] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36433 } } peers { permanent_uuid: "f6a786b358bc487ea9959de22bf75b43" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 33275 } } peers { permanent_uuid: "6e3ed8eb33ab4c0b846162b69aeed22f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38151 } }
I20260923 17:23:47.593583 8366 ts_tablet_manager.cc:1428] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
I20260923 17:23:47.595395 8367 raft_consensus.cc:348] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36433 } } peers { permanent_uuid: "f6a786b358bc487ea9959de22bf75b43" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 33275 } } peers { permanent_uuid: "6e3ed8eb33ab4c0b846162b69aeed22f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38151 } }
I20260923 17:23:47.595907 8367 raft_consensus.cc:374] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:47.596029 8367 raft_consensus.cc:729] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e70fe7b9d3f4b87b553c14b0746e9bc, State: Initialized, Role: FOLLOWER
I20260923 17:23:47.596693 8367 consensus_queue.cc:260] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36433 } } peers { permanent_uuid: "f6a786b358bc487ea9959de22bf75b43" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 33275 } } peers { permanent_uuid: "6e3ed8eb33ab4c0b846162b69aeed22f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38151 } }
I20260923 17:23:47.597733 8368 raft_consensus.cc:348] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36433 } } peers { permanent_uuid: "f6a786b358bc487ea9959de22bf75b43" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 33275 } } peers { permanent_uuid: "6e3ed8eb33ab4c0b846162b69aeed22f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38151 } }
I20260923 17:23:47.598245 8368 raft_consensus.cc:374] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:47.598687 8347 heartbeater.cc:503] Master 127.28.72.254:43017 was elected leader, sending a full tablet report...
I20260923 17:23:47.598712 8368 raft_consensus.cc:729] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3ed8eb33ab4c0b846162b69aeed22f, State: Initialized, Role: FOLLOWER
I20260923 17:23:47.599215 8367 ts_tablet_manager.cc:1428] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260923 17:23:47.599391 8368 consensus_queue.cc:260] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36433 } } peers { permanent_uuid: "f6a786b358bc487ea9959de22bf75b43" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 33275 } } peers { permanent_uuid: "6e3ed8eb33ab4c0b846162b69aeed22f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38151 } }
I20260923 17:23:47.600490 8213 heartbeater.cc:503] Master 127.28.72.254:43017 was elected leader, sending a full tablet report...
I20260923 17:23:47.601179 8368 ts_tablet_manager.cc:1428] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260923 17:23:47.609532 8214 tablet.cc:2369] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:23:47.613840 8372 raft_consensus.cc:482] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:23:47.614176 8372 raft_consensus.cc:504] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36433 } } peers { permanent_uuid: "f6a786b358bc487ea9959de22bf75b43" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 33275 } } peers { permanent_uuid: "6e3ed8eb33ab4c0b846162b69aeed22f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38151 } }
I20260923 17:23:47.616096 8372 leader_election.cc:288] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e70fe7b9d3f4b87b553c14b0746e9bc (127.28.72.195:36433), 6e3ed8eb33ab4c0b846162b69aeed22f (127.28.72.194:38151)
I20260923 17:23:47.625001 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.617227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:45635 (local address 127.28.72.194:38151)
0923 17:23:47.617551 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:47.617558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:47.617787 (+ 229us) server_negotiation.cc:407] Connection header received
0923 17:23:47.618102 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:47.618115 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:47.618174 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:47.618278 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:47.618761 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.619709 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.622439 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.623727 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.623895 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.623973 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:47.624631 (+ 658us) server_negotiation.cc:299] Negotiation successful
0923 17:23:47.624760 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:23:47.625036 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.616505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:33111 (local address 127.28.72.195:36433)
0923 17:23:47.616700 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:47.616704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:47.617100 (+ 396us) server_negotiation.cc:407] Connection header received
0923 17:23:47.617432 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:47.617438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:47.617518 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:47.617596 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:47.618291 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.619373 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.622436 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.623760 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.623903 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.624012 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:47.624598 (+ 586us) server_negotiation.cc:299] Negotiation successful
0923 17:23:47.624729 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:23:47.625242 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.616291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36433 (local address 127.28.72.193:33111)
0923 17:23:47.616977 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:47.616997 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:47.617185 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:47.617774 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:47.617776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:47.617798 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:47.618153 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:47.618162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.619504 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.619508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:47.622263 (+ 2755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:47.622280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.624068 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.624073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:47.624368 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.624425 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 17:23:47.624539 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 17:23:47.624689 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":62,"spinlock_wait_cycles":25472,"thread_start_us":349,"threads_started":1}
I20260923 17:23:47.625264 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.617061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:38151 (local address 127.28.72.193:45635)
0923 17:23:47.617677 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:47.617693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:47.617886 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:47.618320 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:47.618323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:47.618343 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:47.618622 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:47.618629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.620244 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.620250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:47.622285 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:47.622294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.624045 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.624057 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:47.624366 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.624503 (+ 137us) client_negotiation.cc:769] Sending connection context
0923 17:23:47.624583 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:23:47.624689 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":58,"thread_start_us":290,"threads_started":1}
I20260923 17:23:47.626324 8168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9bb554fc614f47b12aae97315273de" candidate_uuid: "f6a786b358bc487ea9959de22bf75b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3ed8eb33ab4c0b846162b69aeed22f" is_pre_election: true
I20260923 17:23:47.626323 8302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9bb554fc614f47b12aae97315273de" candidate_uuid: "f6a786b358bc487ea9959de22bf75b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc" is_pre_election: true
I20260923 17:23:47.626848 8168 raft_consensus.cc:2454] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a786b358bc487ea9959de22bf75b43 in term 0.
I20260923 17:23:47.626890 8302 raft_consensus.cc:2454] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a786b358bc487ea9959de22bf75b43 in term 0.
I20260923 17:23:47.627588 7960 leader_election.cc:302] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e70fe7b9d3f4b87b553c14b0746e9bc, f6a786b358bc487ea9959de22bf75b43; no voters:
I20260923 17:23:47.628012 8372 raft_consensus.cc:2781] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:23:47.628170 8372 raft_consensus.cc:482] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:23:47.628280 8372 raft_consensus.cc:3037] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:47.630470 8372 raft_consensus.cc:504] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36433 } } peers { permanent_uuid: "f6a786b358bc487ea9959de22bf75b43" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 33275 } } peers { permanent_uuid: "6e3ed8eb33ab4c0b846162b69aeed22f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38151 } }
I20260923 17:23:47.631165 8372 leader_election.cc:288] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [CANDIDATE]: Term 1 election: Requested vote from peers 6e70fe7b9d3f4b87b553c14b0746e9bc (127.28.72.195:36433), 6e3ed8eb33ab4c0b846162b69aeed22f (127.28.72.194:38151)
I20260923 17:23:47.631650 8302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9bb554fc614f47b12aae97315273de" candidate_uuid: "f6a786b358bc487ea9959de22bf75b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc"
I20260923 17:23:47.631701 8168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9bb554fc614f47b12aae97315273de" candidate_uuid: "f6a786b358bc487ea9959de22bf75b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3ed8eb33ab4c0b846162b69aeed22f"
I20260923 17:23:47.631863 8302 raft_consensus.cc:3037] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:47.631897 8168 raft_consensus.cc:3037] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:47.634085 8168 raft_consensus.cc:2454] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a786b358bc487ea9959de22bf75b43 in term 1.
I20260923 17:23:47.634559 7960 leader_election.cc:302] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3ed8eb33ab4c0b846162b69aeed22f, f6a786b358bc487ea9959de22bf75b43; no voters:
I20260923 17:23:47.634618 8302 raft_consensus.cc:2454] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a786b358bc487ea9959de22bf75b43 in term 1.
I20260923 17:23:47.634872 8372 raft_consensus.cc:2781] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:47.635442 8372 raft_consensus.cc:686] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [term 1 LEADER]: Becoming Leader. State: Replica: f6a786b358bc487ea9959de22bf75b43, State: Running, Role: LEADER
I20260923 17:23:47.635963 8372 consensus_queue.cc:237] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36433 } } peers { permanent_uuid: "f6a786b358bc487ea9959de22bf75b43" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 33275 } } peers { permanent_uuid: "6e3ed8eb33ab4c0b846162b69aeed22f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38151 } }
I20260923 17:23:47.640856 7856 catalog_manager.cc:5495] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6a786b358bc487ea9959de22bf75b43 (127.28.72.193). New cstate: current_term: 1 leader_uuid: "f6a786b358bc487ea9959de22bf75b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a786b358bc487ea9959de22bf75b43" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 33275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e3ed8eb33ab4c0b846162b69aeed22f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38151 } health_report { overall_health: UNKNOWN } } }
I20260923 17:23:47.679034 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.672866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.28.72.193:33275)
0923 17:23:47.673117 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:47.673125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:47.673148 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:23:47.673255 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:47.673259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:47.673339 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:47.673496 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:23:47.673879 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.674667 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.676778 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.677187 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.677326 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.677879 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:23:47.678447 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:23:47.678613 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:47.678672 (+ 59us) server_negotiation.cc:299] Negotiation successful
0923 17:23:47.678756 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:23:47.688319 8302 raft_consensus.cc:1261] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc [term 1 FOLLOWER]: Refusing update from remote peer f6a786b358bc487ea9959de22bf75b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:23:47.688310 8168 raft_consensus.cc:1261] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f [term 1 FOLLOWER]: Refusing update from remote peer f6a786b358bc487ea9959de22bf75b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:23:47.689373 8377 consensus_queue.cc:1035] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3ed8eb33ab4c0b846162b69aeed22f" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:23:47.689882 8372 consensus_queue.cc:1035] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e70fe7b9d3f4b87b553c14b0746e9bc" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:23:47.704428 8382 mvcc.cc:205] Tried to move back new op lower bound from 7332594596601184256 to 7332594596400521216. Current Snapshot: MvccSnapshot[applied={T|T < 7332594596601184256}]
I20260923 17:23:47.709019 8380 mvcc.cc:205] Tried to move back new op lower bound from 7332594596601184256 to 7332594596400521216. Current Snapshot: MvccSnapshot[applied={T|T < 7332594596601184256 or (T in {7332594596601184256})}]
I20260923 17:23:47.719359 7856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:23:47.719688 7856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:23:47.723323 7856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5040feb0e4934af2995b0531ba3fd27d: Sending DeleteTablet for 3 replicas of tablet 8a9bb554fc614f47b12aae97315273de
I20260923 17:23:47.724627 8005 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9bb554fc614f47b12aae97315273de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:47 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:48644
I20260923 17:23:47.724671 8282 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9bb554fc614f47b12aae97315273de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:47 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:50718
I20260923 17:23:47.724783 8148 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9bb554fc614f47b12aae97315273de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:47 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:47088
I20260923 17:23:47.727449 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 7937
I20260923 17:23:47.756219 8389 tablet_replica.cc:330] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43: stopping tablet replica
I20260923 17:23:47.760388 8389 raft_consensus.cc:2229] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:23:47.762606 8389 raft_consensus.cc:2258] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:47.767417 8389 ts_tablet_manager.cc:1905] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:47.768121 8390 tablet_replica.cc:330] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc: stopping tablet replica
I20260923 17:23:47.769295 8391 tablet_replica.cc:330] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f: stopping tablet replica
I20260923 17:23:47.769508 8390 raft_consensus.cc:2229] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:23:47.769871 8391 raft_consensus.cc:2229] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:23:47.770054 8390 raft_consensus.cc:2258] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:47.770473 8391 raft_consensus.cc:2258] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 23 17:23:47 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:23:47.773015 8391 ts_tablet_manager.cc:1905] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:47.773768 8390 ts_tablet_manager.cc:1905] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:47.778083 8389 ts_tablet_manager.cc:1918] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:47.778309 8389 log.cc:1196] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/8a9bb554fc614f47b12aae97315273de
I20260923 17:23:47.778978 8389 ts_tablet_manager.cc:1939] T 8a9bb554fc614f47b12aae97315273de P f6a786b358bc487ea9959de22bf75b43: Deleting consensus metadata
I20260923 17:23:47.781291 8390 ts_tablet_manager.cc:1918] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:47.781556 8390 log.cc:1196] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/8a9bb554fc614f47b12aae97315273de
I20260923 17:23:47.782122 8390 ts_tablet_manager.cc:1939] T 8a9bb554fc614f47b12aae97315273de P 6e70fe7b9d3f4b87b553c14b0746e9bc: Deleting consensus metadata
I20260923 17:23:47.784260 7843 catalog_manager.cc:4841] TS 6e70fe7b9d3f4b87b553c14b0746e9bc (127.28.72.195:36433): tablet 8a9bb554fc614f47b12aae97315273de (table test-table [id=1e485c6ef7e349eab5bed3e3f3c40123]) successfully deleted
I20260923 17:23:47.785282 7844 catalog_manager.cc:4841] TS f6a786b358bc487ea9959de22bf75b43 (127.28.72.193:33275): tablet 8a9bb554fc614f47b12aae97315273de (table test-table [id=1e485c6ef7e349eab5bed3e3f3c40123]) successfully deleted
I20260923 17:23:47.786386 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.729851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.28.72.193:33275)
0923 17:23:47.730051 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:47.730057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:47.753119 (+ 23062us) server_negotiation.cc:407] Connection header received
0923 17:23:47.753691 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:47.753698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:47.753795 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:47.753886 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:47.756480 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.757807 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.763132 (+ 5325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.763818 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.763923 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.772979 (+ 9056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:47.772997 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:47.773020 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:47.773074 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:47.777730 (+ 4656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:47.778737 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:47.778750 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:47.778754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:47.778980 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:47.779700 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:47.779706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:47.779710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:47.784153 (+ 4443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:47.785777 (+ 1624us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:47.785925 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 17:23:47.786119 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 17:23:47.786360 8391 ts_tablet_manager.cc:1918] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:47.786540 8391 log.cc:1196] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/8a9bb554fc614f47b12aae97315273de
I20260923 17:23:47.787091 8391 ts_tablet_manager.cc:1939] T 8a9bb554fc614f47b12aae97315273de P 6e3ed8eb33ab4c0b846162b69aeed22f: Deleting consensus metadata
I20260923 17:23:47.787226 8065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:23:47.788826 7843 catalog_manager.cc:4841] TS 6e3ed8eb33ab4c0b846162b69aeed22f (127.28.72.194:38151): tablet 8a9bb554fc614f47b12aae97315273de (table test-table [id=1e485c6ef7e349eab5bed3e3f3c40123]) successfully deleted
I20260923 17:23:47.902484 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 7937
I20260923 17:23:47.921116 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 8076
Sep 23 17:23:47 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:23:47.937636 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.921816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.28.72.194:38151)
0923 17:23:47.922004 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:47.922009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:47.922028 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:23:47.922206 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:47.922211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:47.922298 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:47.922444 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:47.923903 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.924721 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.926141 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.926608 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.926718 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.931550 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:47.931575 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:47.931602 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:47.931665 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:47.934742 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:47.935587 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:47.935595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:47.935598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:47.935743 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:47.936105 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:47.936108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:47.936110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:47.936504 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:47.936721 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:47.937062 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 17:23:47.937237 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:23:47.938413 8208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:23:48.030862 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 8076
I20260923 17:23:48.048864 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 8219
Sep 23 17:23:48 dist-test-slave-j0m4 krb5kdc[7381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:23:48.065189 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:48.049766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50728 (local address 127.28.72.195:36433)
0923 17:23:48.050057 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:48.050061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:48.050082 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:23:48.050180 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:48.050183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:48.050257 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:48.050345 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:48.051921 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:48.052657 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:48.054097 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:48.054510 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:48.054622 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:48.059314 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:48.059331 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:48.059348 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:48.059439 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:48.062318 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:48.063106 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:48.063172 (+ 66us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:48.063178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:48.063341 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:48.063749 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:48.063753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:48.063755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:48.064157 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:48.064369 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:48.064745 (+ 376us) server_negotiation.cc:299] Negotiation successful
0923 17:23:48.064915 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260923 17:23:48.065970 8342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:23:48.159936 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 8219
I20260923 17:23:48.176612 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 7825
I20260923 17:23:48.177803 7887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:23:48.294785 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 7825
I20260923 17:23:48.319239 28963 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 17:23:48.655843 28963 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 17:23:48.656101 28963 mini_ranger.cc:66] Stopping Ranger...
I20260923 17:23:49.002849 28963 mini_ranger.cc:68] Stopped Ranger
2026-09-23 17:23:49.003 UTC [7411] LOG: received smart shutdown request
2026-09-23 17:23:49.007 UTC [7411] LOG: background worker "logical replication launcher" (PID 7420) exited with exit code 1
2026-09-23 17:23:49.008 UTC [7415] LOG: shutting down
2026-09-23 17:23:49.240 UTC [7411] LOG: database system is shut down
2026-09-23T17:23:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53054 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:49 dist-test-slave-j0m4 krb5kdc[8405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:49 dist-test-slave-j0m4 krb5kdc[8405](info): set up 2 sockets
Sep 23 17:23:49 dist-test-slave-j0m4 krb5kdc[8405](info): commencing operation
krb5kdc: starting...
W20260923 17:23:50.511722 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:23:50 dist-test-slave-j0m4 krb5kdc[8405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.72.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.72.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/rangeradmin_127.28.72.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/rangeradmin_127.28.72.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.72.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.72.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/rangerlookup_127.28.72.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/rangerlookup_127.28.72.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.72.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.72.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/HTTP_127.28.72.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/HTTP_127.28.72.212@KRBTEST.COM.keytab.
I20260923 17:23:50.893177 28963 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSv3JX5/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/postgres -l logfile start
2026-09-23 17:23:53.779 UTC [8435] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 17:23:53.779 UTC [8435] LOG: listening on IPv4 address "127.28.72.212", port 58521
2026-09-23 17:23:53.783 UTC [8435] LOG: listening on Unix socket "/tmp/.s.PGSQL.58521"
2026-09-23 17:23:53.805 UTC [8438] LOG: database system was shut down at 2026-09-23 17:23:52 UTC
2026-09-23 17:23:53.809 UTC [8435] LOG: database system is ready to accept connections
I20260923 17:23:54.783401 28963 mini_postgres.cc:96] Postgres bound to 58521
2026-09-23 17:23:54.801 UTC [8446] FATAL: database "slave" does not exist
127.28.72.212:58521 - accepting connections
I20260923 17:23:54.802183 28963 mini_ranger.cc:161] Starting Ranger...
I20260923 17:23:54.831861 28963 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 17:23:55.435724 28963 mini_ranger.cc:87] Created ranger Postgres database
I20260923 17:23:55.435853 28963 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-admin
2026-09-23 17:23:55,747 [I] DB FLAVOR :POSTGRES
2026-09-23 17:23:55,748 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:23:55,748 [I] Checking connection..
2026-09-23 17:23:55,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 17:23:56,014 [I] Checking connection passed.
2026-09-23 17:23:56,014 [I] --------- Verifying version history table ---------
2026-09-23 17:23:56,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:23:56,309 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:23:56,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:23:56,567 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:23:56,567 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 17:23:56,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 17:23:56.828 UTC [8510] WARNING: there is no transaction in progress
2026-09-23 17:23:56,835 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 17:23:56,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:23:57,107 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 17:23:57,107 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:23:57,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 17:23:57,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 17:23:57,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j0m4.c.gcp-upstream.internal','N') ;"
2026-09-23 17:23:57,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:23:58,140 [I] Table x_portal_user does not exist in database ranger
2026-09-23 17:23:58,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:23:58,409 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 17:23:58,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:23:58,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:23:58,923 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 17:23:58,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 17:24:00.030 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.049 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.068 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.087 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.109 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.132 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.200 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.208 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.225 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.238 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.249 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.264 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.276 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.284 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.292 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:00.300 UTC [8618] WARNING: there is no transaction in progress
2026-09-23 17:24:01,141 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 17:24:01,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 17:24:01,420 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 17:24:01,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:24:01,682 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 17:24:01,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:24:01,953 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 17:24:01,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:24:02,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:24:02,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j0m4.c.gcp-upstream.internal';"
2026-09-23 17:24:02,757 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 17:24:02,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 17:24:03,019 [I] DB_PATCHES have already been applied
I20260923 17:24:03.025717 28963 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSv3JX5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 17:24:03.025818 28963 mini_ranger.cc:193] Using host: 127.28.72.212
I20260923 17:24:03.041482 28963 mini_ranger.cc:227] Ranger admin URL: http://127.28.72.212:58247
Sep 23, 2026 5:24:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:24:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:24:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 5:24:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 5:24:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 5:24:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 5:24:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 17:24:04.208817 28963 mini_ranger.cc:160] Time spent starting Ranger: real 9.407s user 0.000s sys 0.046s
Sep 23, 2026 5:24:26 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 5:24:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 23, 2026 5:24:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 5:24:26 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 5:24:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 5:24:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 5:24:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 5:24:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 5:24:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 5:24:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 5:24:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 5:24:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 5:24:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 5:24:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 5:24:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 5:24:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 5:24:26 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 5:24:27 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 17:24:28.299836 28963 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.72.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.72.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/rangerkms_127.28.72.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.72.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/rangerkms_127.28.72.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.72.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.72.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.72.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/HTTP_127.28.72.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.72.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/HTTP_127.28.72.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 17:24:28 dist-test-slave-j0m4 krb5kdc[8405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184268, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 17:24:28.493227 28963 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 17:24:28.518028 28963 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 17:24:29.627005 28963 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 17:24:29.627172 28963 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-kms
I20260923 17:24:29.627224 28963 mini_ranger_kms.cc:227] Using postgres at 127.28.72.212:58521
2026-09-23 17:24:30,007 [I] DB FLAVOR :POSTGRES
2026-09-23 17:24:30,008 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:24:30,008 [I] Checking connection
2026-09-23 17:24:30,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 17:24:30,270 [I] connection success
2026-09-23 17:24:30,270 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 17:24:30,271 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 17:24:30,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 17:24:30,523 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 17:24:30,523 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:24:30,523 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 17:24:30,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.72.212:58521/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 17:24:30,807 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 17:24:31.320361 28963 mini_ranger_kms.cc:310] Created kms service
I20260923 17:24:31.525110 28963 mini_ranger_kms.cc:326] Created kudu user
I20260923 17:24:31.578219 28963 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 17:24:31 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184268, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:24:32.000325 28963 mini_ranger_kms.cc:384] Added ranger policy
I20260923 17:24:32.000670 28963 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSv3JX5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSv3JX5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 17:24:32.000751 28963 mini_ranger_kms.cc:240] Using host: 127.28.72.212
I20260923 17:24:32.021227 28963 mini_ranger_kms.cc:276] Ranger KMS PID: 8799
I20260923 17:24:32.021553 28963 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.72.212:47815
Sep 23, 2026 5:24:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:24:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:24:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 5:24:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 5:24:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 5:24:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 17:24:34.317904 28963 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.825s user 0.089s sys 0.310s
I20260923 17:24:34.318166 28963 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 17:24:34.318344 28963 mini_ranger_kms.cc:401] 127.28.72.212:47815/kms/v1/keys
Sep 23 17:24:34 dist-test-slave-j0m4 krb5kdc[8405](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184274, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.72.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 17:24:34 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184274, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.72.212@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
Sep 23 17:24:35 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184274, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.72.212@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
Sep 23 17:24:37 dist-test-slave-j0m4 krb5kdc[8405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:24:37.756335 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:35743
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:35743
--ranger_config_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.72.212:47815/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:38.135841 8849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:38.136157 8849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:38.136315 8849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:38.143790 8849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:38.143919 8849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:38.144001 8849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:38.144093 8849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:38.144188 8849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:38.154691 8849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.72.212:47815/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:35743
--ranger_config_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:35743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:38.156113 8849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:38.157491 8849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:38.170468 8855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:38.170595 8857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:38.171003 8854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:38.171518 8849 server_base.cc:1034] running on GCE node
I20260923 17:24:38.172458 8849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:38.174129 8849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:38.175354 8849 hybrid_clock.cc:648] HybridClock initialized: now 1790184278175312 us; error 44 us; skew 500 ppm
I20260923 17:24:38.175735 8849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:38 dist-test-slave-j0m4 krb5kdc[8405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184278, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:38.186514 8849 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:24:38.188416 8849 webserver.cc:460] Webserver started at http://127.28.72.254:38739/ using document root <none> and password file <none>
I20260923 17:24:38.189146 8849 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:38.189260 8849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:38.189596 8849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:24:38 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184278, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:24:38.257356 8849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "de6905204fc842c8abceee1641cade99"
format_stamp: "Formatted at 2026-09-23 17:24:38 on dist-test-slave-j0m4"
server_key: "5eb8b68e742caab8e344269de24a9bef"
server_key_iv: "36d56356a2221a669615e3c91736916e"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:38.258353 8849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "de6905204fc842c8abceee1641cade99"
format_stamp: "Formatted at 2026-09-23 17:24:38 on dist-test-slave-j0m4"
server_key: "5eb8b68e742caab8e344269de24a9bef"
server_key_iv: "36d56356a2221a669615e3c91736916e"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:38.264679 8849 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 17:24:38.286240 8864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:38.288158 8849 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:24:38.288405 8849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "de6905204fc842c8abceee1641cade99"
format_stamp: "Formatted at 2026-09-23 17:24:38 on dist-test-slave-j0m4"
server_key: "5eb8b68e742caab8e344269de24a9bef"
server_key_iv: "36d56356a2221a669615e3c91736916e"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:38.288663 8849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:38.316848 8849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:38.321300 8849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:38.321652 8849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:38.344115 8849 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:35743
I20260923 17:24:38.344169 8915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:35743 every 8 connection(s)
I20260923 17:24:38.345647 8849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:38.349924 8916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:38.354586 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 8849
I20260923 17:24:38.354892 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:24:38.355671 28963 external_mini_cluster.cc:1428] Setting key 74929ca45e068092c96e0cb7c860b1c5
I20260923 17:24:38.363303 8916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99: Bootstrap starting.
I20260923 17:24:38.367044 8916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:38.368271 8916 log.cc:824] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:38.371647 8916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99: No bootstrap required, opened a new log
Sep 23 17:24:38 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:24:38.382129 8916 raft_consensus.cc:348] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6905204fc842c8abceee1641cade99" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35743 } }
I20260923 17:24:38.382742 8916 raft_consensus.cc:374] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:38.382861 8916 raft_consensus.cc:729] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de6905204fc842c8abceee1641cade99, State: Initialized, Role: FOLLOWER
I20260923 17:24:38.383734 8916 consensus_queue.cc:260] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6905204fc842c8abceee1641cade99" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35743 } }
I20260923 17:24:38.384049 8916 raft_consensus.cc:388] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:38.384230 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:38.358161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.28.72.254:35743)
0923 17:24:38.359081 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:38.359097 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:38.359186 (+ 89us) server_negotiation.cc:407] Connection header received
0923 17:24:38.361774 (+ 2588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:38.361807 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:38.362357 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:38.362871 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:38.365314 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.367400 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.369302 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.370166 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.370351 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:38.374919 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:38.374931 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:38.374953 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:38.375005 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:38.379290 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:38.380130 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:38.380140 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:38.380151 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:38.380389 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:38.380807 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:38.380813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:38.380817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:38.381635 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:38.381954 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:38.382455 (+ 501us) server_negotiation.cc:299] Negotiation successful
0923 17:24:38.383009 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":399,"threads_started":1}
I20260923 17:24:38.384258 8916 raft_consensus.cc:482] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:38.384508 8916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:38.387858 8916 raft_consensus.cc:504] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6905204fc842c8abceee1641cade99" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35743 } }
I20260923 17:24:38.388406 8916 leader_election.cc:302] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de6905204fc842c8abceee1641cade99; no voters:
I20260923 17:24:38.389077 8916 leader_election.cc:288] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:38.389225 8921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:38.390452 8921 raft_consensus.cc:686] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 1 LEADER]: Becoming Leader. State: Replica: de6905204fc842c8abceee1641cade99, State: Running, Role: LEADER
I20260923 17:24:38.391027 8921 consensus_queue.cc:237] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6905204fc842c8abceee1641cade99" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35743 } }
I20260923 17:24:38.391433 8916 sys_catalog.cc:564] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:38.394577 8922 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de6905204fc842c8abceee1641cade99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6905204fc842c8abceee1641cade99" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35743 } } }
I20260923 17:24:38.394793 8923 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de6905204fc842c8abceee1641cade99. Latest consensus state: current_term: 1 leader_uuid: "de6905204fc842c8abceee1641cade99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6905204fc842c8abceee1641cade99" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35743 } } }
I20260923 17:24:38.395138 8922 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:38.395146 8923 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:38.397428 8929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:38.402772 8929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:38.403668 8916 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 17:24:38.421658 8929 catalog_manager.cc:1347] Generated new cluster ID: 5f3bd06c45b54ca38c626a89d24e9f8c
I20260923 17:24:38.421947 8929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:38.436159 8929 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:38.438612 8929 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:38.451813 8929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99: Generated new TSK 0
I20260923 17:24:38.453106 8929 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:24:39.216715 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.72.212:47815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:35743
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:24:39 dist-test-slave-j0m4 krb5kdc[8405](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184279, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 17:24:39.711822 8961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:39.712266 8961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:39.712438 8961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:39.724577 8961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:39.724740 8961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:39.724985 8961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:24:39.741452 8961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.72.212:47815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:35743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:39.743088 8961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:39.744549 8961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:39.762374 8972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:39.769011 8971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:39.769992 8974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:39.770082 8961 server_base.cc:1034] running on GCE node
I20260923 17:24:39.771024 8961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:39.776386 8961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:39.777798 8961 hybrid_clock.cc:648] HybridClock initialized: now 1790184279777674 us; error 120 us; skew 500 ppm
I20260923 17:24:39.778272 8961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:39 dist-test-slave-j0m4 krb5kdc[8405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184279, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:39.782894 8961 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:24:39.785818 8961 webserver.cc:460] Webserver started at http://127.28.72.193:42293/ using document root <none> and password file <none>
I20260923 17:24:39.786939 8961 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:39.787135 8961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:39.787537 8961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:24:39 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184279, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:24:39.819095 8961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "78c03fd1f5414537bc44b6e4c4a40947"
format_stamp: "Formatted at 2026-09-23 17:24:39 on dist-test-slave-j0m4"
server_key: "faa4e20ccf5228fd8846042e32987239"
server_key_iv: "8859ad84ecf1d46b0d7a5e06c7d20047"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:39.820506 8961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "78c03fd1f5414537bc44b6e4c4a40947"
format_stamp: "Formatted at 2026-09-23 17:24:39 on dist-test-slave-j0m4"
server_key: "faa4e20ccf5228fd8846042e32987239"
server_key_iv: "8859ad84ecf1d46b0d7a5e06c7d20047"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:39.836064 8961 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.004s sys 0.009s
I20260923 17:24:39.865377 8981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:39.867666 8961 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.006s
I20260923 17:24:39.867888 8961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "78c03fd1f5414537bc44b6e4c4a40947"
format_stamp: "Formatted at 2026-09-23 17:24:39 on dist-test-slave-j0m4"
server_key: "faa4e20ccf5228fd8846042e32987239"
server_key_iv: "8859ad84ecf1d46b0d7a5e06c7d20047"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:39.868181 8961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:39.896090 8961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:39.902060 8961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:39.902503 8961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:39.903821 8961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:39.905709 8961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:39.905848 8961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:39.906015 8961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:39.906118 8961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:24:39 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184279, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:24:40.071599 8961 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:38411
I20260923 17:24:40.071617 9093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:38411 every 8 connection(s)
I20260923 17:24:40.073153 8961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:24:40.075210 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 8961
I20260923 17:24:40.075444 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:24:40.076488 28963 external_mini_cluster.cc:1428] Setting key d08ec826e57802d7a26c2e0418b25813
Sep 23 17:24:40 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184279, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
I20260923 17:24:40.113775 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.079441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:40325 (local address 127.28.72.254:35743)
0923 17:24:40.091749 (+ 12308us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.091756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.094363 (+ 2607us) server_negotiation.cc:407] Connection header received
0923 17:24:40.096321 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.096326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.096439 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.096571 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:40.098730 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.099705 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.101546 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.101995 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.102143 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.106068 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:40.106075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:40.106078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:40.106116 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:40.109743 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.110906 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.110912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.110916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.111090 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.111708 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.111714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.111720 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.112071 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:40.112236 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.113088 (+ 852us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.113352 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12135,"thread_start_us":12319,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
I20260923 17:24:40.115234 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.093034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35743 (local address 127.28.72.193:40325)
0923 17:24:40.094143 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:40.094205 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:40.096010 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:40.096889 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:40.096902 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:40.097643 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:40.098524 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.098545 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.099920 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.099926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.101353 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.101384 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.102291 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.102300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.102572 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.103484 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:40.103511 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:40.105777 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:40.109971 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.109983 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.110007 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.110740 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.111248 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.111251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.111254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.111538 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.112229 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:40.112238 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:40.112633 (+ 395us) client_negotiation.cc:769] Sending connection context
0923 17:24:40.112997 (+ 364us) client_negotiation.cc:240] Negotiation successful
0923 17:24:40.113692 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":398,"threads_started":1}
I20260923 17:24:40.117261 9094 heartbeater.cc:347] Connected to a master server at 127.28.72.254:35743
I20260923 17:24:40.117741 9094 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:40.118759 9094 heartbeater.cc:511] Master 127.28.72.254:35743 requested a full tablet report, sending...
I20260923 17:24:40.125154 8881 ts_manager.cc:194] Registered new tserver with Master: 78c03fd1f5414537bc44b6e4c4a40947 (127.28.72.193:38411)
I20260923 17:24:40.127964 8881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:40325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:24:40.155634 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.72.212:47815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:35743
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:24:40 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184279, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
W20260923 17:24:40.647931 9100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:40.648401 9100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:40.648571 9100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:40.659713 9100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:40.659884 9100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:40.660074 9100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:24:40.675529 9100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.72.212:47815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:35743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:40.677273 9100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:40.679029 9100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:40.696215 9115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:40.696301 9114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:40.700594 9117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:40.701279 9100 server_base.cc:1034] running on GCE node
I20260923 17:24:40.702157 9100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:40.703361 9100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:40.705545 9100 hybrid_clock.cc:648] HybridClock initialized: now 1790184280705489 us; error 41 us; skew 500 ppm
I20260923 17:24:40.705938 9100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:40 dist-test-slave-j0m4 krb5kdc[8405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184280, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:40.710032 9100 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:24:40.712035 9100 webserver.cc:460] Webserver started at http://127.28.72.194:38555/ using document root <none> and password file <none>
I20260923 17:24:40.712987 9100 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:40.713124 9100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:40.713479 9100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:24:40 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184280, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:24:40.730383 9100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "3c6f600d7748477797156035ca563b6b"
format_stamp: "Formatted at 2026-09-23 17:24:40 on dist-test-slave-j0m4"
server_key: "215ade0e7b4587e75f11dd61fb917d35"
server_key_iv: "42f6c0a105d3517dc592d78b970bc0dc"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:40.731570 9100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "3c6f600d7748477797156035ca563b6b"
format_stamp: "Formatted at 2026-09-23 17:24:40 on dist-test-slave-j0m4"
server_key: "215ade0e7b4587e75f11dd61fb917d35"
server_key_iv: "42f6c0a105d3517dc592d78b970bc0dc"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:40.738998 9100 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260923 17:24:40.758267 9124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:40.760260 9100 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260923 17:24:40.760461 9100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "3c6f600d7748477797156035ca563b6b"
format_stamp: "Formatted at 2026-09-23 17:24:40 on dist-test-slave-j0m4"
server_key: "215ade0e7b4587e75f11dd61fb917d35"
server_key_iv: "42f6c0a105d3517dc592d78b970bc0dc"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:40.760713 9100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:40.795225 9100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:40.801054 9100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:40.801409 9100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:40.802661 9100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:40.804554 9100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:40.804648 9100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:40.804755 9100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:40.804816 9100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:40.845414 9100 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:37533
I20260923 17:24:40.845386 9236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:37533 every 8 connection(s)
I20260923 17:24:40.847138 9100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:24:40.852385 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 9100
I20260923 17:24:40.852576 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:24:40.853106 28963 external_mini_cluster.cc:1428] Setting key 0b70f424516fadcd753bf74bd1bb571f
Sep 23 17:24:40 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184280, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:24:40.879196 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.850881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35743 (local address 127.28.72.194:38851)
0923 17:24:40.851811 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:40.851864 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:40.854041 (+ 2177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:40.855003 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:40.855016 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:40.855672 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:40.856543 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.856579 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.858267 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.858273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.860553 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.860581 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.861794 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.861801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.862115 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.863009 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:40.863040 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:40.866216 (+ 3176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:40.870659 (+ 4443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.870673 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.870699 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.871546 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.874984 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.874990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.874994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.875391 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.876229 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:40.876238 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:40.876748 (+ 510us) client_negotiation.cc:769] Sending connection context
0923 17:24:40.877074 (+ 326us) client_negotiation.cc:240] Negotiation successful
0923 17:24:40.877678 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":445,"threads_started":1}
I20260923 17:24:40.880442 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.850145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:38851 (local address 127.28.72.254:35743)
0923 17:24:40.850726 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.850732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.852410 (+ 1678us) server_negotiation.cc:407] Connection header received
0923 17:24:40.854492 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.854498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.854575 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.854687 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:40.856773 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.858055 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.860752 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.861522 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.861699 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.866577 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:40.866586 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:40.866592 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:40.866639 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:40.870449 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.874601 (+ 4152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.874608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.874612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.874811 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.875555 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.875561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.875564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.875976 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:40.876146 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.879767 (+ 3621us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.880055 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":309,"threads_started":1}
I20260923 17:24:40.882915 9237 heartbeater.cc:347] Connected to a master server at 127.28.72.254:35743
I20260923 17:24:40.883358 9237 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
I20260923 17:24:40.884416 9237 heartbeater.cc:511] Master 127.28.72.254:35743 requested a full tablet report, sending...
I20260923 17:24:40.886552 8881 ts_manager.cc:194] Registered new tserver with Master: 3c6f600d7748477797156035ca563b6b (127.28.72.194:37533)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
I20260923 17:24:40.887902 8881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:38851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:24:40.926980 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.72.212:47815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:35743
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:24:41.132560 9094 heartbeater.cc:503] Master 127.28.72.254:35743 was elected leader, sending a full tablet report...
W20260923 17:24:41.291231 9243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:41.291625 9243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:41.291792 9243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:41.300117 9243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:41.300293 9243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:41.300472 9243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:24:41.312207 9243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.72.212:47815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:35743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:41.313725 9243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:41.315158 9243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:41.329991 9249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:41.330127 9248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:41.330057 9251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:41.330973 9243 server_base.cc:1034] running on GCE node
I20260923 17:24:41.331733 9243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:41.332820 9243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:41.334064 9243 hybrid_clock.cc:648] HybridClock initialized: now 1790184281333999 us; error 68 us; skew 500 ppm
I20260923 17:24:41.334436 9243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:41 dist-test-slave-j0m4 krb5kdc[8405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184281, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:41.338264 9243 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:24:41.340075 9243 webserver.cc:460] Webserver started at http://127.28.72.195:44463/ using document root <none> and password file <none>
I20260923 17:24:41.340866 9243 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:41.341025 9243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:41.341349 9243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:24:41 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184281, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:24:41.358809 9243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "e4bf9fafd0e94025be944d8e8cb94785"
format_stamp: "Formatted at 2026-09-23 17:24:41 on dist-test-slave-j0m4"
server_key: "3a1e16ff597f82f3cccfa1703445abf5"
server_key_iv: "900fa38bbf2515b987a73544d443f9fb"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:41.359860 9243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "e4bf9fafd0e94025be944d8e8cb94785"
format_stamp: "Formatted at 2026-09-23 17:24:41 on dist-test-slave-j0m4"
server_key: "3a1e16ff597f82f3cccfa1703445abf5"
server_key_iv: "900fa38bbf2515b987a73544d443f9fb"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:41.366307 9243 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 17:24:41.385623 9258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:41.387629 9243 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 17:24:41.387861 9243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "e4bf9fafd0e94025be944d8e8cb94785"
format_stamp: "Formatted at 2026-09-23 17:24:41 on dist-test-slave-j0m4"
server_key: "3a1e16ff597f82f3cccfa1703445abf5"
server_key_iv: "900fa38bbf2515b987a73544d443f9fb"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:41.388118 9243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:41.410324 9243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:41.414917 9243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:41.415326 9243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:41.416419 9243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:41.417898 9243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:41.418006 9243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:41.418162 9243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:41.418236 9243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:41.453614 9243 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:38437
I20260923 17:24:41.453714 9370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:38437 every 8 connection(s)
I20260923 17:24:41.455230 9243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:24:41.463501 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 9243
I20260923 17:24:41.463706 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:24:41.464175 28963 external_mini_cluster.cc:1428] Setting key 10343cd57355a8d9e6e58b5a1e6f81df
Sep 23 17:24:41 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184281, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:24:41.479346 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.458165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:51127 (local address 127.28.72.254:35743)
0923 17:24:41.458657 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:41.458661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:41.459733 (+ 1072us) server_negotiation.cc:407] Connection header received
0923 17:24:41.461557 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:41.461562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:41.461664 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:41.461787 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:41.463466 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.464520 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.466764 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.467372 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.467521 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.472058 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:41.472064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:41.472068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:41.472117 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:41.475371 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:41.476410 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:41.476416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:41.476418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:41.476564 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:41.477172 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:41.477177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:41.477179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:41.477550 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:41.477678 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:41.478645 (+ 967us) server_negotiation.cc:299] Negotiation successful
0923 17:24:41.478916 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":225,"threads_started":1}
I20260923 17:24:41.480607 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.458718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35743 (local address 127.28.72.195:51127)
0923 17:24:41.459558 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:41.459604 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:41.461267 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:41.462038 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:41.462048 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:41.462480 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:41.463231 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:41.463264 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.464707 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.464713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:41.466572 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:41.466587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.467642 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.467653 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:41.467936 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.468873 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:41.468903 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:41.471727 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:41.475595 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:41.475605 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:41.475623 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:41.476180 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:41.476745 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:41.476749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:41.476752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:41.477030 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:41.477703 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:41.477710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:41.478127 (+ 417us) client_negotiation.cc:769] Sending connection context
0923 17:24:41.478543 (+ 416us) client_negotiation.cc:240] Negotiation successful
0923 17:24:41.479188 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":337,"threads_started":1}
I20260923 17:24:41.482326 9371 heartbeater.cc:347] Connected to a master server at 127.28.72.254:35743
I20260923 17:24:41.482615 9371 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:41.483431 9371 heartbeater.cc:511] Master 127.28.72.254:35743 requested a full tablet report, sending...
I20260923 17:24:41.485067 8881 ts_manager.cc:194] Registered new tserver with Master: e4bf9fafd0e94025be944d8e8cb94785 (127.28.72.195:38437)
I20260923 17:24:41.485949 8881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:51127
I20260923 17:24:41.493927 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:24:41.495762 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 8961
Sep 23 17:24:41 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:24:41.513872 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.496921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.28.72.193:38411)
0923 17:24:41.497772 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:41.497779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:41.497807 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:24:41.497863 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:41.497869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:41.498084 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:41.498216 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:41.499814 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.500905 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.502501 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.503090 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.503281 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.507977 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:41.507990 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:41.508006 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:41.508047 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:41.510881 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:41.511638 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:41.511645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:41.511647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:41.511811 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:41.512213 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:41.512217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:41.512220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:41.512648 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:41.512897 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:41.513306 (+ 409us) server_negotiation.cc:299] Negotiation successful
0923 17:24:41.513584 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":522,"threads_started":1}
I20260923 17:24:41.515332 9089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:41.604048 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 8961
I20260923 17:24:41.621526 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 9100
Sep 23 17:24:41 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:24:41.639312 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.622635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.28.72.194:37533)
0923 17:24:41.623226 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:41.623234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:41.623259 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:24:41.623333 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:41.623339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:41.623562 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:41.623699 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:41.624944 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.626005 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.627423 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.627972 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.628079 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.632805 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:41.632819 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:41.632836 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:41.632884 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:41.636232 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:41.637076 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:41.637084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:41.637086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:41.637221 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:41.637663 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:41.637667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:41.637669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:41.638058 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:41.638360 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:41.638722 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 17:24:41.638948 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":308,"threads_started":1}
I20260923 17:24:41.640736 9232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:41.727002 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 9100
I20260923 17:24:41.748616 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 9243
Sep 23 17:24:41 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:24:41.767263 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.749624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.28.72.195:38437)
0923 17:24:41.750270 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:41.750277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:41.750307 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:24:41.750367 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:41.750374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:41.750604 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:41.750781 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:41.752255 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.753309 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.755039 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.755785 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.755944 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.760704 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:41.760720 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:41.760736 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:41.760774 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:41.764026 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:41.764868 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:41.764884 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:41.764886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:41.765022 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:41.765475 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:41.765479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:41.765481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:41.765848 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:41.766188 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:41.766530 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 17:24:41.766806 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":342,"threads_started":1}
I20260923 17:24:41.768586 9366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:41.853253 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 9243
I20260923 17:24:41.869457 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 8849
I20260923 17:24:41.870792 8911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:41.984194 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 8849
I20260923 17:24:42.007968 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:35743
--webserver_interface=127.28.72.254
--webserver_port=38739
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:35743
--ranger_config_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.72.212:47815/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.72.212:47815/kms with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:42.351402 9383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:42.351716 9383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:42.351806 9383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:42.359179 9383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:42.359309 9383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:42.359357 9383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:42.359407 9383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:42.359455 9383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:42.369867 9383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.72.212:47815/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:35743
--ranger_config_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:35743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=38739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:42.371348 9383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:42.372668 9383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:42.385552 9389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:42.385623 9388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:42.385774 9391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:42.386883 9383 server_base.cc:1034] running on GCE node
I20260923 17:24:42.387972 9383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:42.389824 9383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:42.391155 9383 hybrid_clock.cc:648] HybridClock initialized: now 1790184282391055 us; error 67 us; skew 500 ppm
I20260923 17:24:42.391602 9383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:42 dist-test-slave-j0m4 krb5kdc[8405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184282, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:42.395081 9383 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:24:42.397239 9383 webserver.cc:460] Webserver started at http://127.28.72.254:38739/ using document root <none> and password file <none>
I20260923 17:24:42.398010 9383 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:42.398181 9383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:24:42 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184282, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:24:42.504629 9383 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:24:42.508528 9399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:42.510330 9383 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:24:42.510531 9383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "de6905204fc842c8abceee1641cade99"
format_stamp: "Formatted at 2026-09-23 17:24:38 on dist-test-slave-j0m4"
server_key: "5eb8b68e742caab8e344269de24a9bef"
server_key_iv: "36d56356a2221a669615e3c91736916e"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:42.511935 9383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:42.532167 9383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:42.537396 9383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:42.537796 9383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:42.557814 9383 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:35743
I20260923 17:24:42.557839 9450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:35743 every 8 connection(s)
I20260923 17:24:42.559491 9383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:42.564778 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 9383
I20260923 17:24:42.565702 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.72.212:47815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:38411
--local_ip_for_outbound_sockets=127.28.72.193
--tserver_master_addrs=127.28.72.254:35743
--webserver_port=42293
--webserver_interface=127.28.72.193
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.72.212:47815/kms with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:24:42.567493 9451 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:24:42.571369 9451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99: Bootstrap starting.
I20260923 17:24:42.592448 9451 log.cc:824] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:42.602053 9451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:24:42.602674 9451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99: Bootstrap complete.
I20260923 17:24:42.612195 9451 raft_consensus.cc:348] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6905204fc842c8abceee1641cade99" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35743 } }
I20260923 17:24:42.613185 9451 raft_consensus.cc:729] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de6905204fc842c8abceee1641cade99, State: Initialized, Role: FOLLOWER
I20260923 17:24:42.613996 9451 consensus_queue.cc:260] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6905204fc842c8abceee1641cade99" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35743 } }
I20260923 17:24:42.614282 9451 raft_consensus.cc:388] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:42.614432 9451 raft_consensus.cc:482] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:42.614625 9451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:24:42.619652 9451 raft_consensus.cc:504] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6905204fc842c8abceee1641cade99" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35743 } }
I20260923 17:24:42.620399 9451 leader_election.cc:302] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de6905204fc842c8abceee1641cade99; no voters:
I20260923 17:24:42.621131 9451 leader_election.cc:288] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:24:42.621352 9455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:24:42.623359 9455 raft_consensus.cc:686] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [term 2 LEADER]: Becoming Leader. State: Replica: de6905204fc842c8abceee1641cade99, State: Running, Role: LEADER
I20260923 17:24:42.623706 9451 sys_catalog.cc:564] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:42.623966 9455 consensus_queue.cc:237] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6905204fc842c8abceee1641cade99" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35743 } }
I20260923 17:24:42.628715 9457 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de6905204fc842c8abceee1641cade99. Latest consensus state: current_term: 2 leader_uuid: "de6905204fc842c8abceee1641cade99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6905204fc842c8abceee1641cade99" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35743 } } }
I20260923 17:24:42.629211 9457 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:42.629866 9456 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "de6905204fc842c8abceee1641cade99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6905204fc842c8abceee1641cade99" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35743 } } }
I20260923 17:24:42.630210 9456 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:42.639847 9464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:42.646310 9464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:42.647902 9464 catalog_manager.cc:1259] Loaded cluster ID: 5f3bd06c45b54ca38c626a89d24e9f8c
I20260923 17:24:42.648036 9464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:42.652197 9464 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:42.653856 9464 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99: Loaded TSK: 0
I20260923 17:24:42.655570 9464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:24:42.660714 9451 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260923 17:24:42.993389 9453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:42.993770 9453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:42.993942 9453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:43.006332 9453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:43.006527 9453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:43.006727 9453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:24:43.021148 9453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.72.212:47815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:38411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=42293
--tserver_master_addrs=127.28.72.254:35743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:43.022780 9453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:43.024575 9453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:43.065549 9490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:43.071161 9489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:43.072085 9492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:43.076064 9453 server_base.cc:1034] running on GCE node
I20260923 17:24:43.076992 9453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:43.078421 9453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:43.079682 9453 hybrid_clock.cc:648] HybridClock initialized: now 1790184283079612 us; error 87 us; skew 500 ppm
I20260923 17:24:43.080123 9453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:43 dist-test-slave-j0m4 krb5kdc[8405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:43.084594 9453 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:24:43.097774 9453 webserver.cc:460] Webserver started at http://127.28.72.193:42293/ using document root <none> and password file <none>
I20260923 17:24:43.098963 9453 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:43.099170 9453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:24:43 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:24:43.138362 9453 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.002s sys 0.011s
I20260923 17:24:43.142581 9500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:43.144739 9453 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:24:43.144981 9453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "78c03fd1f5414537bc44b6e4c4a40947"
format_stamp: "Formatted at 2026-09-23 17:24:39 on dist-test-slave-j0m4"
server_key: "faa4e20ccf5228fd8846042e32987239"
server_key_iv: "8859ad84ecf1d46b0d7a5e06c7d20047"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:43.146469 9453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:43.179596 9453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:43.193126 9453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:43.193594 9453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:43.207499 9453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:43.214049 9453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:43.214210 9453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:43.214370 9453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:43.214460 9453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:43.310501 9453 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:38411
I20260923 17:24:43.310542 9613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:38411 every 8 connection(s)
I20260923 17:24:43.312157 9453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:24:43.316154 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 9453
I20260923 17:24:43.317271 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.72.212:47815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:37533
--local_ip_for_outbound_sockets=127.28.72.194
--tserver_master_addrs=127.28.72.254:35743
--webserver_port=38555
--webserver_interface=127.28.72.194
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.72.212:47815/kms with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:24:43 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:24:43.352602 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.316148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:47617 (local address 127.28.72.254:35743)
0923 17:24:43.331736 (+ 15588us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.331758 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.331813 (+ 55us) server_negotiation.cc:407] Connection header received
0923 17:24:43.332292 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.332312 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.332891 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.333291 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:43.334908 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.336380 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.338004 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.338839 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.339106 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.343236 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:43.343248 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:43.343270 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:43.343335 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:43.346647 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.347589 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.347597 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.347602 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.347768 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.348713 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.348718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.348727 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.349244 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:43.349556 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.350437 (+ 881us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.350882 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15314,"thread_start_us":15037,"threads_started":1}
I20260923 17:24:43.358886 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.316445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35743 (local address 127.28.72.193:47617)
0923 17:24:43.328251 (+ 11806us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.328322 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.330379 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.333450 (+ 3071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.333461 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.334065 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:43.334672 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.334691 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.336566 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.336572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.337824 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.337837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.339178 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.339187 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.339527 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.340451 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:43.340476 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:43.342838 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:43.346830 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.346840 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.346858 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.347418 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.348342 (+ 924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.348355 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.348358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.348572 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.349492 (+ 920us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:43.349504 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:43.349885 (+ 381us) client_negotiation.cc:769] Sending connection context
0923 17:24:43.350179 (+ 294us) client_negotiation.cc:240] Negotiation successful
0923 17:24:43.357423 (+ 7244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11384,"thread_start_us":15110,"threads_started":1}
I20260923 17:24:43.363236 9614 heartbeater.cc:347] Connected to a master server at 127.28.72.254:35743
I20260923 17:24:43.363633 9614 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:43.364645 9614 heartbeater.cc:511] Master 127.28.72.254:35743 requested a full tablet report, sending...
W20260923 17:24:43.368096 9614 heartbeater.cc:644] Failed to heartbeat to 127.28.72.254:35743 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:24:43.368351 9614 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:43.368639 9614 heartbeater.cc:511] Master 127.28.72.254:35743 requested a full tablet report, sending...
I20260923 17:24:43.369810 9614 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:43.370126 9614 heartbeater.cc:511] Master 127.28.72.254:35743 requested a full tablet report, sending...
W20260923 17:24:43.371290 9614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:24:43.594954 9625 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260923 17:24:43.764376 9618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:43.764719 9618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:43.764896 9618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:43.772552 9618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:43.772725 9618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:43.772943 9618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:24:43.783974 9618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.72.212:47815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:37533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=38555
--tserver_master_addrs=127.28.72.254:35743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:43.786084 9618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:43.787997 9618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:43.806036 9633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:43.805943 9632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:43.813460 9635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:43.815418 9618 server_base.cc:1034] running on GCE node
I20260923 17:24:43.816362 9618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:43.817888 9618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:43.819236 9618 hybrid_clock.cc:648] HybridClock initialized: now 1790184283819148 us; error 65 us; skew 500 ppm
I20260923 17:24:43.819679 9618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:43 dist-test-slave-j0m4 krb5kdc[8405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:43.824345 9618 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:24:43.828846 9618 webserver.cc:460] Webserver started at http://127.28.72.194:38555/ using document root <none> and password file <none>
I20260923 17:24:43.829914 9618 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:43.830096 9618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:24:43 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:24:43.853242 9618 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260923 17:24:43.859233 9643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:43.861032 9618 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260923 17:24:43.861260 9618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "3c6f600d7748477797156035ca563b6b"
format_stamp: "Formatted at 2026-09-23 17:24:40 on dist-test-slave-j0m4"
server_key: "215ade0e7b4587e75f11dd61fb917d35"
server_key_iv: "42f6c0a105d3517dc592d78b970bc0dc"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:43.862452 9618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:43.914423 9618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:43.920377 9618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:43.920800 9618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:43.929685 9618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:43.932116 9618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:43.932252 9618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:43.932436 9618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:43.932521 9618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:43.974457 9618 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:37533
I20260923 17:24:43.974519 9757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:37533 every 8 connection(s)
I20260923 17:24:43.976578 9618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:24:43.976966 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 9618
I20260923 17:24:43.977741 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.72.212:47815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:38437
--local_ip_for_outbound_sockets=127.28.72.195
--tserver_master_addrs=127.28.72.254:35743
--webserver_port=44463
--webserver_interface=127.28.72.195
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.72.212:47815/kms with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:24:44 dist-test-slave-j0m4 krb5kdc[8405](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 17:24:44 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:24:44.025892 9761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.986703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35743 (local address 127.28.72.194:52937)
0923 17:24:44.000667 (+ 13964us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:44.000725 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:44.003162 (+ 2437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:44.004438 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:44.004450 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:44.005074 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:44.005889 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.005914 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.007541 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.007546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.009504 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.009522 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.011094 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.011104 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.011521 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.012754 (+ 1233us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:44.012796 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:44.016049 (+ 3253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:44.020955 (+ 4906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:44.020966 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:44.020986 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:44.021608 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:44.022057 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:44.022062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:44.022066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:44.022347 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:44.023072 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:44.023082 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:44.023579 (+ 497us) client_negotiation.cc:769] Sending connection context
0923 17:24:44.023889 (+ 310us) client_negotiation.cc:240] Negotiation successful
0923 17:24:44.024488 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13606,"thread_start_us":16556,"threads_started":1}
I20260923 17:24:44.026902 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.986479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:52937 (local address 127.28.72.254:35743)
0923 17:24:43.995709 (+ 9230us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.995717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:44.000920 (+ 5203us) server_negotiation.cc:407] Connection header received
0923 17:24:44.003740 (+ 2820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:44.003746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:44.003844 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:44.003960 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:44.006128 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.007333 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.009712 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.010514 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.010674 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.016408 (+ 5734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:44.016418 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:44.016423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:44.016477 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:44.020708 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:44.021760 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:44.021766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:44.021770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:44.021930 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:44.022481 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:44.022487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:44.022490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:44.022836 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:44.022997 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.026290 (+ 3293us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.026558 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9071,"thread_start_us":8927,"threads_started":1}
I20260923 17:24:44.029054 9758 heartbeater.cc:347] Connected to a master server at 127.28.72.254:35743
I20260923 17:24:44.029385 9758 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:44.030300 9758 heartbeater.cc:511] Master 127.28.72.254:35743 requested a full tablet report, sending...
I20260923 17:24:44.032434 9416 ts_manager.cc:194] Registered new tserver with Master: 3c6f600d7748477797156035ca563b6b (127.28.72.194:37533)
I20260923 17:24:44.034744 9416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:52937
I20260923 17:24:44.371447 9614 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:44.371961 9614 heartbeater.cc:511] Master 127.28.72.254:35743 requested a full tablet report, sending...
I20260923 17:24:44.373318 9416 ts_manager.cc:194] Registered new tserver with Master: 78c03fd1f5414537bc44b6e4c4a40947 (127.28.72.193:38411)
I20260923 17:24:44.374635 9416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:47617
W20260923 17:24:44.481724 9759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:44.482263 9759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:44.482533 9759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:44.494516 9759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:44.494762 9759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:44.495026 9759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:24:44.508595 9759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.72.212:47815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:38437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=44463
--tserver_master_addrs=127.28.72.254:35743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
Sep 23 17:24:44 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:24:44.510365 9759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:44.512156 9759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:44.550498 9769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:44.551568 9768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:44.575430 9759 server_base.cc:1034] running on GCE node
W20260923 17:24:44.578006 9772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:44.578984 9759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:44.580886 9759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:44.584553 9759 hybrid_clock.cc:648] HybridClock initialized: now 1790184284584373 us; error 150 us; skew 500 ppm
I20260923 17:24:44.585057 9759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:44 dist-test-slave-j0m4 krb5kdc[8405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184284, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:44.589570 9759 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:24:44.608780 9759 webserver.cc:460] Webserver started at http://127.28.72.195:44463/ using document root <none> and password file <none>
I20260923 17:24:44.609944 9759 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:44.610193 9759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:24:44 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184284, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:24:44.640420 9759 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260923 17:24:44.644153 9779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:44.646390 9759 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:24:44.646817 9759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "e4bf9fafd0e94025be944d8e8cb94785"
format_stamp: "Formatted at 2026-09-23 17:24:41 on dist-test-slave-j0m4"
server_key: "3a1e16ff597f82f3cccfa1703445abf5"
server_key_iv: "900fa38bbf2515b987a73544d443f9fb"
server_key_version: "kuduclusterkey@0"
I20260923 17:24:44.648082 9759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:44.667418 9759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:44.673422 9759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:44.673749 9759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:44.675169 9759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:44.677100 9759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:44.677239 9759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:44.677368 9759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:44.677459 9759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:44.718549 9759 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:38437
I20260923 17:24:44.718513 9891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:38437 every 8 connection(s)
I20260923 17:24:44.720630 9759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:24:44.727396 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 9759
Sep 23 17:24:44 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184284, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:24:44.769526 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.728657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.28.72.254:35743)
0923 17:24:44.747509 (+ 18852us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:44.747515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:44.747538 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:24:44.747595 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:44.747603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:44.747695 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:44.747825 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:44.750438 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.751838 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.754069 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.754983 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.759232 (+ 4249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.759565 (+ 333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:44.759574 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:44.759580 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:44.759622 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:44.764585 (+ 4963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:44.765756 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:44.765763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:44.765767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:44.765937 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:44.766491 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:44.766496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:44.766501 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:44.766965 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:44.768576 (+ 1611us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.768715 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.768993 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":18704,"spinlock_wait_cycles":1408,"thread_start_us":10703,"threads_started":1}
I20260923 17:24:44.770272 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.723835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:54033 (local address 127.28.72.254:35743)
0923 17:24:44.740879 (+ 17044us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:44.740890 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:44.743715 (+ 2825us) server_negotiation.cc:407] Connection header received
0923 17:24:44.746720 (+ 3005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:44.746727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:44.746850 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:44.746939 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:44.749658 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.750967 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.753950 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.754831 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.754965 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.760075 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:44.760085 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:44.760092 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:44.760148 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:44.765301 (+ 5153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:44.766491 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:44.766496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:44.766501 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:44.766677 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:44.767437 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:44.767444 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:44.767448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:44.767792 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:44.768028 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.769299 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.769548 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":15673,"spinlock_wait_cycles":3456,"thread_start_us":24152,"threads_started":1}
I20260923 17:24:44.773177 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.724775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35743 (local address 127.28.72.195:54033)
0923 17:24:44.743522 (+ 18747us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:44.743582 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:44.746334 (+ 2752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:44.747603 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:44.747617 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:44.748363 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:44.749402 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.749436 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.751215 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.751222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.753742 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.753771 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.755382 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.755391 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.755741 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.756536 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:44.756577 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:44.759765 (+ 3188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:44.765533 (+ 5768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:44.765544 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:44.765566 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:44.766322 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:44.766826 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:44.766832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:44.766836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:44.767266 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:44.768187 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:44.768196 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:44.768740 (+ 544us) client_negotiation.cc:769] Sending connection context
0923 17:24:44.771112 (+ 2372us) client_negotiation.cc:240] Negotiation successful
0923 17:24:44.771800 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18378,"thread_start_us":18072,"threads_started":1}
I20260923 17:24:44.774823 9892 heartbeater.cc:347] Connected to a master server at 127.28.72.254:35743
I20260923 17:24:44.775250 9892 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:44.776060 9892 heartbeater.cc:511] Master 127.28.72.254:35743 requested a full tablet report, sending...
I20260923 17:24:44.777761 9416 ts_manager.cc:194] Registered new tserver with Master: e4bf9fafd0e94025be944d8e8cb94785 (127.28.72.195:38437)
I20260923 17:24:44.778651 9416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:54033
I20260923 17:24:44.786149 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:24:44.806318 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.790669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.28.72.254:35743)
0923 17:24:44.790937 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:44.790947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:44.791740 (+ 793us) server_negotiation.cc:407] Connection header received
0923 17:24:44.792029 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:44.792035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:44.792133 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:44.792251 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:44.793921 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.794784 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.796412 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.796864 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.797046 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.800180 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:44.800187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:44.800192 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:44.800240 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:44.803093 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:44.803995 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:44.804001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:44.804005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:44.804153 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:44.804655 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:44.804660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:44.804663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:44.804975 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:44.805252 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.805712 (+ 460us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.805972 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:24:44.811928 9416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:24:44.822376 9416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 23 17:24:44 dist-test-slave-j0m4 krb5kdc[8405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for HTTP/127.28.72.212@KRBTEST.COM
I20260923 17:24:44.899488 9913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.878221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38411 (local address 127.0.0.1:41158)
0923 17:24:44.890449 (+ 12228us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:44.890473 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:44.890595 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:44.891725 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:44.891730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:44.891769 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:44.892225 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.892239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.894202 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.894207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.896300 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.896316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.898429 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.898435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.898651 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.898703 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 17:24:44.898852 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 17:24:44.899060 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12061,"mutex_wait_us":27,"threads_started":1}
I20260923 17:24:44.900396 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.872626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41158 (local address 127.28.72.193:38411)
0923 17:24:44.880764 (+ 8138us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:44.880772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:44.890755 (+ 9983us) server_negotiation.cc:407] Connection header received
0923 17:24:44.890882 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:44.890894 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:44.891234 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:44.891467 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:44.892450 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.894037 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.896495 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.897902 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.898035 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.898265 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.899753 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.899945 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7958,"thread_start_us":7741,"threads_started":1}
I20260923 17:24:44.900831 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.857235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:38437 (local address 127.0.0.1:46212)
0923 17:24:44.869556 (+ 12321us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:44.869623 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:44.869905 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:44.880775 (+ 10870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:44.880787 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:44.880845 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:44.881328 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.881355 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.890591 (+ 9236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.890597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.893182 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.893198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.899996 (+ 6798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.900004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.900191 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.900234 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 17:24:44.900324 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 17:24:44.900482 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11919,"mutex_wait_us":1,"thread_start_us":14540,"threads_started":1}
I20260923 17:24:44.901815 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.860634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:37533 (local address 127.0.0.1:57480)
0923 17:24:44.872173 (+ 11539us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:44.872202 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:44.883174 (+ 10972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:44.888279 (+ 5105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:44.888285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:44.888319 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:44.888698 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.888711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.893362 (+ 4651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.893368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.895576 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.895591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.901126 (+ 5535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.901131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.901246 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.901268 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 17:24:44.901385 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 17:24:44.901518 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11331,"thread_start_us":8317,"threads_started":1}
I20260923 17:24:44.901961 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.860959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.28.72.194:37533)
0923 17:24:44.882608 (+ 21649us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:44.882616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:44.882648 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:24:44.883400 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:44.883407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:44.887453 (+ 4046us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:44.887724 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:44.888946 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.890411 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.897985 (+ 7574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.899520 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.899649 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.899834 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.901458 (+ 1624us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.901645 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21471,"thread_start_us":22076,"threads_started":1}
I20260923 17:24:44.902653 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.857728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46212 (local address 127.28.72.195:38437)
0923 17:24:44.879367 (+ 21639us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:44.879381 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:44.879435 (+ 54us) server_negotiation.cc:407] Connection header received
0923 17:24:44.879610 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:44.879623 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:44.880058 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:44.880333 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:44.888058 (+ 7725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.889864 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.895780 (+ 5916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.897463 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.897594 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.897850 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.902100 (+ 4250us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.902286 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21347,"thread_start_us":24939,"threads_started":1}
I20260923 17:24:44.906581 9693 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9093fa57f54d6ca4593f1f651f48e8 (DEFAULT_TABLE table=test-table [id=015aaf77dd5443628c595f3337b31d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:44.908119 9693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9093fa57f54d6ca4593f1f651f48e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:44.908914 9827 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9093fa57f54d6ca4593f1f651f48e8 (DEFAULT_TABLE table=test-table [id=015aaf77dd5443628c595f3337b31d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:44.908919 9549 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9093fa57f54d6ca4593f1f651f48e8 (DEFAULT_TABLE table=test-table [id=015aaf77dd5443628c595f3337b31d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:44.910318 9827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9093fa57f54d6ca4593f1f651f48e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:44.910342 9549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9093fa57f54d6ca4593f1f651f48e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:44.933837 9914 tablet_bootstrap.cc:492] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b: Bootstrap starting.
I20260923 17:24:44.937378 9914 tablet_bootstrap.cc:654] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:44.938566 9914 log.cc:824] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:44.946527 9914 tablet_bootstrap.cc:492] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b: No bootstrap required, opened a new log
I20260923 17:24:44.946867 9914 ts_tablet_manager.cc:1397] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20260923 17:24:44.950259 9915 tablet_bootstrap.cc:492] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785: Bootstrap starting.
I20260923 17:24:44.954993 9917 tablet_bootstrap.cc:492] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947: Bootstrap starting.
I20260923 17:24:44.959244 9917 tablet_bootstrap.cc:654] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:44.959959 9914 raft_consensus.cc:348] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bf9fafd0e94025be944d8e8cb94785" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 38437 } } peers { permanent_uuid: "78c03fd1f5414537bc44b6e4c4a40947" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38411 } } peers { permanent_uuid: "3c6f600d7748477797156035ca563b6b" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 37533 } }
I20260923 17:24:44.960464 9914 raft_consensus.cc:374] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:44.960580 9914 raft_consensus.cc:729] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6f600d7748477797156035ca563b6b, State: Initialized, Role: FOLLOWER
I20260923 17:24:44.961092 9917 log.cc:824] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:44.961392 9914 consensus_queue.cc:260] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bf9fafd0e94025be944d8e8cb94785" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 38437 } } peers { permanent_uuid: "78c03fd1f5414537bc44b6e4c4a40947" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38411 } } peers { permanent_uuid: "3c6f600d7748477797156035ca563b6b" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 37533 } }
I20260923 17:24:44.969973 9758 heartbeater.cc:503] Master 127.28.72.254:35743 was elected leader, sending a full tablet report...
I20260923 17:24:44.970284 9914 ts_tablet_manager.cc:1428] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b: Time spent starting tablet: real 0.023s user 0.012s sys 0.008s
I20260923 17:24:44.973023 9917 tablet_bootstrap.cc:492] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947: No bootstrap required, opened a new log
I20260923 17:24:44.973502 9917 ts_tablet_manager.cc:1397] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.004s
I20260923 17:24:44.984764 9915 tablet_bootstrap.cc:654] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:44.984392 9917 raft_consensus.cc:348] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bf9fafd0e94025be944d8e8cb94785" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 38437 } } peers { permanent_uuid: "78c03fd1f5414537bc44b6e4c4a40947" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38411 } } peers { permanent_uuid: "3c6f600d7748477797156035ca563b6b" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 37533 } }
I20260923 17:24:44.985548 9917 raft_consensus.cc:374] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:44.985709 9917 raft_consensus.cc:729] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c03fd1f5414537bc44b6e4c4a40947, State: Initialized, Role: FOLLOWER
I20260923 17:24:44.986150 9915 log.cc:824] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:44.986647 9917 consensus_queue.cc:260] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bf9fafd0e94025be944d8e8cb94785" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 38437 } } peers { permanent_uuid: "78c03fd1f5414537bc44b6e4c4a40947" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38411 } } peers { permanent_uuid: "3c6f600d7748477797156035ca563b6b" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 37533 } }
I20260923 17:24:44.991652 9614 heartbeater.cc:503] Master 127.28.72.254:35743 was elected leader, sending a full tablet report...
I20260923 17:24:44.993594 9917 ts_tablet_manager.cc:1428] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947: Time spent starting tablet: real 0.020s user 0.017s sys 0.000s
I20260923 17:24:44.998152 9915 tablet_bootstrap.cc:492] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785: No bootstrap required, opened a new log
I20260923 17:24:44.998564 9915 ts_tablet_manager.cc:1397] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.000s
W20260923 17:24:44.998919 9762 tablet.cc:2369] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:45.009701 9915 raft_consensus.cc:348] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bf9fafd0e94025be944d8e8cb94785" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 38437 } } peers { permanent_uuid: "78c03fd1f5414537bc44b6e4c4a40947" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38411 } } peers { permanent_uuid: "3c6f600d7748477797156035ca563b6b" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 37533 } }
I20260923 17:24:45.010430 9915 raft_consensus.cc:374] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:45.010607 9915 raft_consensus.cc:729] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4bf9fafd0e94025be944d8e8cb94785, State: Initialized, Role: FOLLOWER
I20260923 17:24:45.011432 9915 consensus_queue.cc:260] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bf9fafd0e94025be944d8e8cb94785" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 38437 } } peers { permanent_uuid: "78c03fd1f5414537bc44b6e4c4a40947" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38411 } } peers { permanent_uuid: "3c6f600d7748477797156035ca563b6b" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 37533 } }
I20260923 17:24:45.027055 9918 raft_consensus.cc:482] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:45.027498 9918 raft_consensus.cc:504] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bf9fafd0e94025be944d8e8cb94785" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 38437 } } peers { permanent_uuid: "78c03fd1f5414537bc44b6e4c4a40947" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38411 } } peers { permanent_uuid: "3c6f600d7748477797156035ca563b6b" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 37533 } }
I20260923 17:24:45.028939 9892 heartbeater.cc:503] Master 127.28.72.254:35743 was elected leader, sending a full tablet report...
I20260923 17:24:45.029826 9915 ts_tablet_manager.cc:1428] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785: Time spent starting tablet: real 0.031s user 0.021s sys 0.008s
I20260923 17:24:45.029824 9918 leader_election.cc:288] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4bf9fafd0e94025be944d8e8cb94785 (127.28.72.195:38437), 78c03fd1f5414537bc44b6e4c4a40947 (127.28.72.193:38411)
I20260923 17:24:45.044183 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.035247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:34243 (local address 127.28.72.193:38411)
0923 17:24:45.035459 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:45.035466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:45.036035 (+ 569us) server_negotiation.cc:407] Connection header received
0923 17:24:45.036227 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:45.036233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:45.036352 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:45.036457 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:45.037150 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.038294 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.041747 (+ 3453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.042991 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.043226 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:45.043307 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:45.043770 (+ 463us) server_negotiation.cc:299] Negotiation successful
0923 17:24:45.043923 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:24:45.044229 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.035249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38411 (local address 127.28.72.194:34243)
0923 17:24:45.035893 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:45.035913 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:45.036062 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:45.036529 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:45.036535 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:45.036566 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:45.036948 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:45.036963 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.038495 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.038500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:45.041548 (+ 3048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:45.041567 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.043269 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.043276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:45.043564 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:45.043603 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 17:24:45.043712 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 17:24:45.043862 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":341,"threads_started":1}
I20260923 17:24:45.045653 9569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9093fa57f54d6ca4593f1f651f48e8" candidate_uuid: "3c6f600d7748477797156035ca563b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c03fd1f5414537bc44b6e4c4a40947" is_pre_election: true
I20260923 17:24:45.046324 9569 raft_consensus.cc:2454] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6f600d7748477797156035ca563b6b in term 0.
I20260923 17:24:45.047214 9647 leader_election.cc:302] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6f600d7748477797156035ca563b6b, 78c03fd1f5414537bc44b6e4c4a40947; no voters:
I20260923 17:24:45.047832 9918 raft_consensus.cc:2781] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:24:45.047986 9918 raft_consensus.cc:482] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:24:45.048079 9918 raft_consensus.cc:3037] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:45.051544 9918 raft_consensus.cc:504] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bf9fafd0e94025be944d8e8cb94785" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 38437 } } peers { permanent_uuid: "78c03fd1f5414537bc44b6e4c4a40947" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38411 } } peers { permanent_uuid: "3c6f600d7748477797156035ca563b6b" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 37533 } }
I20260923 17:24:45.052783 9918 leader_election.cc:288] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [CANDIDATE]: Term 1 election: Requested vote from peers e4bf9fafd0e94025be944d8e8cb94785 (127.28.72.195:38437), 78c03fd1f5414537bc44b6e4c4a40947 (127.28.72.193:38411)
I20260923 17:24:45.053575 9569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9093fa57f54d6ca4593f1f651f48e8" candidate_uuid: "3c6f600d7748477797156035ca563b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c03fd1f5414537bc44b6e4c4a40947"
I20260923 17:24:45.053836 9569 raft_consensus.cc:3037] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:45.057374 9569 raft_consensus.cc:2454] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6f600d7748477797156035ca563b6b in term 1.
I20260923 17:24:45.057838 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.031350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:57039 (local address 127.28.72.195:38437)
0923 17:24:45.031536 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:45.031543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:45.049337 (+ 17794us) server_negotiation.cc:407] Connection header received
0923 17:24:45.050031 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:45.050037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:45.050123 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:45.050190 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:45.051437 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.052596 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.055397 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.056605 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.056744 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:45.056821 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:45.057443 (+ 622us) server_negotiation.cc:299] Negotiation successful
0923 17:24:45.057587 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 17:24:45.058209 9647 leader_election.cc:302] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6f600d7748477797156035ca563b6b, 78c03fd1f5414537bc44b6e4c4a40947; no voters:
I20260923 17:24:45.058643 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.030309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:38437 (local address 127.28.72.194:57039)
0923 17:24:45.049206 (+ 18897us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:45.049232 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:45.049485 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:45.050425 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:45.050430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:45.050457 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:45.050853 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:45.050868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.052971 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.052978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:45.055183 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:45.055200 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.057029 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.057035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:45.057221 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:45.057259 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 17:24:45.058103 (+ 844us) client_negotiation.cc:240] Negotiation successful
0923 17:24:45.058253 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4467,"thread_start_us":4288,"threads_started":1}
I20260923 17:24:45.059820 9847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9093fa57f54d6ca4593f1f651f48e8" candidate_uuid: "3c6f600d7748477797156035ca563b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4bf9fafd0e94025be944d8e8cb94785" is_pre_election: true
I20260923 17:24:45.060413 9847 raft_consensus.cc:2454] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6f600d7748477797156035ca563b6b in term 0.
I20260923 17:24:45.061384 9847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9093fa57f54d6ca4593f1f651f48e8" candidate_uuid: "3c6f600d7748477797156035ca563b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4bf9fafd0e94025be944d8e8cb94785"
I20260923 17:24:45.061583 9847 raft_consensus.cc:3037] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:45.063257 9918 raft_consensus.cc:2781] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:45.064990 9847 raft_consensus.cc:2454] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6f600d7748477797156035ca563b6b in term 1.
W20260923 17:24:45.066839 9615 tablet.cc:2369] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:45.067389 9918 raft_consensus.cc:686] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [term 1 LEADER]: Becoming Leader. State: Replica: 3c6f600d7748477797156035ca563b6b, State: Running, Role: LEADER
I20260923 17:24:45.068085 9918 consensus_queue.cc:237] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bf9fafd0e94025be944d8e8cb94785" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 38437 } } peers { permanent_uuid: "78c03fd1f5414537bc44b6e4c4a40947" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38411 } } peers { permanent_uuid: "3c6f600d7748477797156035ca563b6b" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 37533 } }
I20260923 17:24:45.082312 9416 catalog_manager.cc:5495] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c6f600d7748477797156035ca563b6b (127.28.72.194). New cstate: current_term: 1 leader_uuid: "3c6f600d7748477797156035ca563b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bf9fafd0e94025be944d8e8cb94785" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 38437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c03fd1f5414537bc44b6e4c4a40947" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c6f600d7748477797156035ca563b6b" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 37533 } health_report { overall_health: HEALTHY } } }
I20260923 17:24:45.124586 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.116724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.28.72.194:37533)
0923 17:24:45.116934 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:45.116941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:45.117115 (+ 174us) server_negotiation.cc:407] Connection header received
0923 17:24:45.117390 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:45.117396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:45.117511 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:45.117703 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:45.118226 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.119437 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.121527 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.122194 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.122329 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:45.123077 (+ 748us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:45.123874 (+ 797us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:45.124036 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:45.124179 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 17:24:45.124265 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:24:45.134961 9569 raft_consensus.cc:1261] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947 [term 1 FOLLOWER]: Refusing update from remote peer 3c6f600d7748477797156035ca563b6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:45.135001 9847 raft_consensus.cc:1261] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785 [term 1 FOLLOWER]: Refusing update from remote peer 3c6f600d7748477797156035ca563b6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:45.136300 9925 consensus_queue.cc:1035] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4bf9fafd0e94025be944d8e8cb94785" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 38437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:45.136941 9918 consensus_queue.cc:1035] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c03fd1f5414537bc44b6e4c4a40947" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:45.166693 9930 mvcc.cc:205] Tried to move back new op lower bound from 7332594831901327360 to 7332594831644819456. Current Snapshot: MvccSnapshot[applied={T|T < 7332594831901327360}]
I20260923 17:24:45.169003 9929 mvcc.cc:205] Tried to move back new op lower bound from 7332594831901327360 to 7332594831644819456. Current Snapshot: MvccSnapshot[applied={T|T < 7332594831901327360}]
I20260923 17:24:45.193756 9416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:45.194115 9416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:45.198973 9416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de6905204fc842c8abceee1641cade99: Sending DeleteTablet for 3 replicas of tablet 2a9093fa57f54d6ca4593f1f651f48e8
I20260923 17:24:45.200719 9827 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a9093fa57f54d6ca4593f1f651f48e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:45 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:46212
I20260923 17:24:45.201763 9693 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a9093fa57f54d6ca4593f1f651f48e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:45 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:57480
I20260923 17:24:45.203392 9549 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a9093fa57f54d6ca4593f1f651f48e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:45 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:41158
I20260923 17:24:45.211499 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 9453
I20260923 17:24:45.236965 9945 tablet_replica.cc:330] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785: stopping tablet replica
I20260923 17:24:45.237722 9946 tablet_replica.cc:330] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b: stopping tablet replica
I20260923 17:24:45.238348 9947 tablet_replica.cc:330] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947: stopping tablet replica
I20260923 17:24:45.238850 9947 raft_consensus.cc:2229] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:45.238852 9945 raft_consensus.cc:2229] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:45.239486 9947 raft_consensus.cc:2258] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:45.241431 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.213655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.28.72.193:38411)
0923 17:24:45.213893 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:45.213902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:45.213925 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:24:45.214003 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:45.214011 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:45.214118 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:45.214236 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:45.219003 (+ 4767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.220506 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.224039 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.224999 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.225156 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:45.229562 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:45.229589 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:45.229617 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:45.229679 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:45.234799 (+ 5120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:45.236019 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:45.236035 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:45.236041 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:45.236247 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:45.237975 (+ 1728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:45.237981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:45.237986 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:45.239200 (+ 1214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:45.240554 (+ 1354us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:45.240763 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 17:24:45.241040 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:24:45.242250 9945 raft_consensus.cc:2258] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:45.242300 9609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:45.240056 9946 raft_consensus.cc:2229] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:24:45.243386 9946 raft_consensus.cc:2258] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:45.245844 9945 ts_tablet_manager.cc:1905] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:45.248852 9946 ts_tablet_manager.cc:1905] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:45.263011 9945 ts_tablet_manager.cc:1918] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:24:45.265287 9945 log.cc:1196] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/2a9093fa57f54d6ca4593f1f651f48e8
I20260923 17:24:45.266606 9945 ts_tablet_manager.cc:1939] T 2a9093fa57f54d6ca4593f1f651f48e8 P e4bf9fafd0e94025be944d8e8cb94785: Deleting consensus metadata
I20260923 17:24:45.267959 9946 ts_tablet_manager.cc:1918] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:24:45.268262 9946 log.cc:1196] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/2a9093fa57f54d6ca4593f1f651f48e8
I20260923 17:24:45.268879 9946 ts_tablet_manager.cc:1939] T 2a9093fa57f54d6ca4593f1f651f48e8 P 3c6f600d7748477797156035ca563b6b: Deleting consensus metadata
I20260923 17:24:45.270484 9403 catalog_manager.cc:4841] TS e4bf9fafd0e94025be944d8e8cb94785 (127.28.72.195:38437): tablet 2a9093fa57f54d6ca4593f1f651f48e8 (table test-table [id=015aaf77dd5443628c595f3337b31d0d]) successfully deleted
I20260923 17:24:45.273033 9402 catalog_manager.cc:4841] TS 3c6f600d7748477797156035ca563b6b (127.28.72.194:37533): tablet 2a9093fa57f54d6ca4593f1f651f48e8 (table test-table [id=015aaf77dd5443628c595f3337b31d0d]) successfully deleted
I20260923 17:24:45.429508 9947 ts_tablet_manager.cc:1905] T 2a9093fa57f54d6ca4593f1f651f48e8 P 78c03fd1f5414537bc44b6e4c4a40947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:45.435722 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 9453
W20260923 17:24:45.460821 9403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.72.193:38411 (error 108)
I20260923 17:24:45.461849 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 9618
I20260923 17:24:45.475720 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.462802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38411 (local address 127.0.0.1:41180)
0923 17:24:45.475349 (+ 12547us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:45.475518 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12386,"threads_started":1}
W20260923 17:24:45.476249 9403 catalog_manager.cc:4584] TS 78c03fd1f5414537bc44b6e4c4a40947 (127.28.72.193:38411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a9093fa57f54d6ca4593f1f651f48e8: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38411: connect: Connection refused (error 111)
I20260923 17:24:45.483222 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.463233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.28.72.194:37533)
0923 17:24:45.463423 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:45.463429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:45.463454 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:24:45.463525 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:45.463530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:45.463617 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:45.463730 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:45.465643 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.467031 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.469253 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.470040 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.470208 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:45.474227 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:45.474249 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:45.474305 (+ 56us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:45.474367 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:45.479418 (+ 5051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:45.480485 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:45.480496 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:45.480500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:45.480726 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:45.481208 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:45.481213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:45.481217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:45.481712 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:45.482064 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:45.482521 (+ 457us) server_negotiation.cc:299] Negotiation successful
0923 17:24:45.482775 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:24:45.484251 9753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:45.530176 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.529599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38411 (local address 127.0.0.1:41188)
0923 17:24:45.529860 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:45.529996 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260923 17:24:45.531770 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.531206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38411 (local address 127.0.0.1:41202)
0923 17:24:45.531484 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:45.531582 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 17:24:45.592523 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.591975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38411 (local address 127.0.0.1:48530)
0923 17:24:45.592227 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:45.592341 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260923 17:24:45.593771 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.593336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38411 (local address 127.0.0.1:48544)
0923 17:24:45.593509 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:45.593597 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 17:24:45.609862 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 9618
I20260923 17:24:45.630896 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 9759
I20260923 17:24:45.646901 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.631838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34366 (local address 127.28.72.195:38437)
0923 17:24:45.632713 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:45.632718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:45.632733 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:24:45.632788 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:45.632792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:45.632851 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:45.633002 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:45.634742 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.635603 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.637100 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.637512 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.637617 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:45.640518 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:45.640535 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:45.640551 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:45.640592 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:45.643772 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:45.644605 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:45.644612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:45.644615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:45.644756 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:45.645183 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:45.645188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:45.645191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:45.645569 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:45.645923 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:45.646212 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 17:24:45.646445 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":616,"threads_started":1}
I20260923 17:24:45.647990 9887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:45.674872 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.674248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38411 (local address 127.0.0.1:48554)
0923 17:24:45.674606 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:45.674705 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260923 17:24:45.676218 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.675716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38411 (local address 127.0.0.1:48564)
0923 17:24:45.676000 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:45.676071 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260923 17:24:45.741087 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 9759
I20260923 17:24:45.757917 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 9383
I20260923 17:24:45.759353 9446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:45.874449 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.873716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38411 (local address 127.0.0.1:48580)
0923 17:24:45.874075 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:45.874183 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260923 17:24:45.874684 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 9383
I20260923 17:24:45.900707 28963 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 17:24:46.237594 28963 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 17:24:46.237861 28963 mini_ranger.cc:66] Stopping Ranger...
I20260923 17:24:46.584540 28963 mini_ranger.cc:68] Stopped Ranger
2026-09-23 17:24:46.584 UTC [8435] LOG: received smart shutdown request
2026-09-23 17:24:46.588 UTC [8435] LOG: background worker "logical replication launcher" (PID 8444) exited with exit code 1
2026-09-23 17:24:46.588 UTC [8439] LOG: shutting down
2026-09-23 17:24:46.823 UTC [8435] LOG: database system is shut down
2026-09-23T17:24:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57572 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:24:47 dist-test-slave-j0m4 krb5kdc[9963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:24:47 dist-test-slave-j0m4 krb5kdc[9963](info): set up 2 sockets
Sep 23 17:24:47 dist-test-slave-j0m4 krb5kdc[9963](info): commencing operation
krb5kdc: starting...
W20260923 17:24:48.087579 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:24:48 dist-test-slave-j0m4 krb5kdc[9963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:24:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:24:48.339963 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:35607
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:35607,127.28.72.253:37403,127.28.72.252:43677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:48.700553 9976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:48.700863 9976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:48.700953 9976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:48.708551 9976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:48.708675 9976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:48.708722 9976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:48.708773 9976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:48.708819 9976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:48.719135 9976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:35607,127.28.72.253:37403,127.28.72.252:43677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:35607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:48.720521 9976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:48.721837 9976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:48.734266 9982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:48.734510 9984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:48.734704 9981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:48.735445 9976 server_base.cc:1034] running on GCE node
I20260923 17:24:48.736290 9976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:48.737798 9976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:48.739073 9976 hybrid_clock.cc:648] HybridClock initialized: now 1790184288739018 us; error 47 us; skew 500 ppm
I20260923 17:24:48.739563 9976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:48 dist-test-slave-j0m4 krb5kdc[9963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184288, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:48.747771 9976 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:24:48.749619 9976 webserver.cc:460] Webserver started at http://127.28.72.254:34481/ using document root <none> and password file <none>
I20260923 17:24:48.750403 9976 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:48.750560 9976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:48.750928 9976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:48.754091 9976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "a97f835816814b879d35f2c7707b2c1b"
format_stamp: "Formatted at 2026-09-23 17:24:48 on dist-test-slave-j0m4"
server_key: "59291c342768e297a102aeb5c852b10c"
server_key_iv: "17d8aea6fa00ea897849e860a1467eb5"
server_key_version: "encryptionkey@0"
I20260923 17:24:48.754974 9976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "a97f835816814b879d35f2c7707b2c1b"
format_stamp: "Formatted at 2026-09-23 17:24:48 on dist-test-slave-j0m4"
server_key: "59291c342768e297a102aeb5c852b10c"
server_key_iv: "17d8aea6fa00ea897849e860a1467eb5"
server_key_version: "encryptionkey@0"
I20260923 17:24:48.760752 9976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:24:48.764973 9991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:48.766549 9976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:24:48.766789 9976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "a97f835816814b879d35f2c7707b2c1b"
format_stamp: "Formatted at 2026-09-23 17:24:48 on dist-test-slave-j0m4"
server_key: "59291c342768e297a102aeb5c852b10c"
server_key_iv: "17d8aea6fa00ea897849e860a1467eb5"
server_key_version: "encryptionkey@0"
I20260923 17:24:48.767071 9976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:48.791235 9976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:48.796321 9976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:48.796661 9976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:48.817770 9976 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:35607
I20260923 17:24:48.817788 10042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:35607 every 8 connection(s)
I20260923 17:24:48.819248 9976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:48.823237 10043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:48.826133 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 9976
I20260923 17:24:48.826397 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:24:48.827091 28963 external_mini_cluster.cc:1428] Setting key 7303361e0d42c8bd8b28849fe2789b26
I20260923 17:24:48.830328 10043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:24:48 dist-test-slave-j0m4 krb5kdc[9963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184288, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.72.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.253@KRBTEST.COM" created.
I20260923 17:24:48.858842 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:48.833736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35607 (local address 127.0.0.1:51688)
0923 17:24:48.834727 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:48.834779 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:48.837806 (+ 3027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:48.838843 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:48.838853 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:48.839839 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:48.840392 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:48.840424 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:48.842147 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:48.842156 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:48.844261 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:48.844288 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:48.845464 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:48.845474 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:48.845725 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:48.846465 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:48.846508 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:48.849515 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:48.853956 (+ 4441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:48.853967 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:48.853977 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:48.854429 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:48.854956 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:48.854959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:48.854961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:48.855186 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:48.856276 (+ 1090us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:48.856283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:48.856483 (+ 200us) client_negotiation.cc:769] Sending connection context
0923 17:24:48.856733 (+ 250us) client_negotiation.cc:240] Negotiation successful
0923 17:24:48.857327 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":419,"threads_started":1}
I20260923 17:24:48.858868 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:48.833736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.28.72.254:35607)
0923 17:24:48.835799 (+ 2063us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:48.835805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:48.835854 (+ 49us) server_negotiation.cc:407] Connection header received
0923 17:24:48.838290 (+ 2436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:48.838297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:48.838566 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:48.838759 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:48.840652 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:48.841965 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:48.844484 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:48.845166 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:48.845327 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:48.849792 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:48.849802 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:48.849821 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:48.849861 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:48.853763 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:48.854599 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:48.854612 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:48.854618 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:48.854830 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:48.855317 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:48.855325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:48.855331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:48.856119 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:48.856360 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:48.856942 (+ 582us) server_negotiation.cc:299] Negotiation successful
0923 17:24:48.857542 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":226,"server-negotiator.queue_time_us":1855,"spinlock_wait_cycles":63744,"thread_start_us":1609,"threads_started":1}
I20260923 17:24:48.862437 10043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:24:48.863544 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:48.863090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:37403 (local address 127.0.0.1:53450)
0923 17:24:48.863295 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:48.863421 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260923 17:24:48.863807 9993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
I20260923 17:24:48.865556 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:48.865100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:37403 (local address 127.0.0.1:53458)
0923 17:24:48.865270 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:48.865431 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 17:24:48.865811 10043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:37403: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.253.keytab.
Entry for principal kudu/127.28.72.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.253.keytab.
I20260923 17:24:48.883966 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.253:37403
--webserver_interface=127.28.72.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.253.keytab
--principal=kudu/127.28.72.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:35607,127.28.72.253:37403,127.28.72.252:43677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:24:48.915217 10043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } attempt: 1
I20260923 17:24:48.916466 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:48.915993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:37403 (local address 127.0.0.1:53466)
0923 17:24:48.916205 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:48.916315 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 17:24:48.917585 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:48.917200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:37403 (local address 127.0.0.1:53474)
0923 17:24:48.917378 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:48.917451 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 17:24:48.917781 10043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:37403: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
I20260923 17:24:48.986079 10043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } attempt: 2
I20260923 17:24:48.987439 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:48.986842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:37403 (local address 127.0.0.1:53480)
0923 17:24:48.987182 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:48.987285 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 17:24:48.988638 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:48.988213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:37403 (local address 127.0.0.1:53488)
0923 17:24:48.988388 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:48.988489 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 17:24:48.988843 10043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:37403: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
I20260923 17:24:49.080127 10043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } attempt: 3
I20260923 17:24:49.081374 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.080926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:37403 (local address 127.0.0.1:53498)
0923 17:24:49.081140 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.081238 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260923 17:24:49.082686 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.082162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:37403 (local address 127.0.0.1:53510)
0923 17:24:49.082324 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.082507 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 17:24:49.082890 10043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:37403: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
I20260923 17:24:49.226147 10043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } attempt: 4
I20260923 17:24:49.227459 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.226915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:37403 (local address 127.0.0.1:53520)
0923 17:24:49.227233 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.227324 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260923 17:24:49.228608 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.228198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:37403 (local address 127.0.0.1:53524)
0923 17:24:49.228367 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.228478 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260923 17:24:49.228812 10043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:37403: Network error: Client connection negotiation failed: client connection to 127.28.72.253:37403: connect: Connection refused (error 111)
W20260923 17:24:49.229593 10050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:49.229926 10050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:49.230080 10050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:49.237740 10050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:49.237914 10050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:49.238029 10050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:49.238133 10050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:49.238215 10050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:49.249132 10050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:35607,127.28.72.253:37403,127.28.72.252:43677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.253.keytab
--principal=kudu/127.28.72.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.253:37403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.72.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:49.250660 10050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:49.252099 10050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:49.264422 10055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:49.264497 10056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:49.264766 10058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:49.265792 10050 server_base.cc:1034] running on GCE node
I20260923 17:24:49.266623 10050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:49.268148 10050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:49.269389 10050 hybrid_clock.cc:648] HybridClock initialized: now 1790184289269351 us; error 51 us; skew 500 ppm
I20260923 17:24:49.269760 10050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:49 dist-test-slave-j0m4 krb5kdc[9963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:49.273411 10050 init.cc:377] Logged in from keytab as kudu/127.28.72.253@KRBTEST.COM (short username kudu)
I20260923 17:24:49.274991 10050 webserver.cc:460] Webserver started at http://127.28.72.253:36507/ using document root <none> and password file <none>
I20260923 17:24:49.275756 10050 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:49.275866 10050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:49.276144 10050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:49.279328 10050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/data/instance:
uuid: "8270a990a5454e82a94cf5fdd6da8e46"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-j0m4"
server_key: "db10cbb85e7386cb11e6f645dc7b8f07"
server_key_iv: "d8d58eb2936bc4c904e8ab609fcb268e"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.280159 10050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/wal/instance:
uuid: "8270a990a5454e82a94cf5fdd6da8e46"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-j0m4"
server_key: "db10cbb85e7386cb11e6f645dc7b8f07"
server_key_iv: "d8d58eb2936bc4c904e8ab609fcb268e"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.285770 10050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:24:49.289511 10065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:49.291105 10050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:24:49.291461 10050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/wal
uuid: "8270a990a5454e82a94cf5fdd6da8e46"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-j0m4"
server_key: "db10cbb85e7386cb11e6f645dc7b8f07"
server_key_iv: "d8d58eb2936bc4c904e8ab609fcb268e"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.291661 10050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:49.313285 10050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:49.318032 10050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:49.318348 10050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:49.338570 10050 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.253:37403
I20260923 17:24:49.338632 10116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.253:37403 every 8 connection(s)
I20260923 17:24:49.340090 10050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/data/info.pb
I20260923 17:24:49.344241 10117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:49.348088 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 10050
I20260923 17:24:49.348313 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-1/wal/instance
I20260923 17:24:49.348778 28963 external_mini_cluster.cc:1428] Setting key f13ae1927459ace13bccdc6ff651a52d
I20260923 17:24:49.351257 10117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:24:49 dist-test-slave-j0m4 krb5kdc[9963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.253@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.72.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.252@KRBTEST.COM" created.
I20260923 17:24:49.377417 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.354098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51694 (local address 127.28.72.254:35607)
0923 17:24:49.354298 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.354304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:49.356138 (+ 1834us) server_negotiation.cc:407] Connection header received
0923 17:24:49.358544 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:49.358549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:49.358626 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:49.358726 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:49.361053 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.361788 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.364472 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.364934 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.365039 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.369634 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:49.369644 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:49.369651 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:49.369705 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:49.373727 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.374661 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.374666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.374669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.374802 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.375318 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.375323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.375325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.375643 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:49.375768 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:49.376685 (+ 917us) server_negotiation.cc:299] Negotiation successful
0923 17:24:49.376973 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:24:49.378522 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.354832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35607 (local address 127.0.0.1:51694)
0923 17:24:49.355899 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.355957 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:49.358209 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:49.359030 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:49.359045 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:49.359942 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:49.360811 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.360841 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.361990 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.361999 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.364281 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.364302 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.365194 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.365204 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.365539 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.366353 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:49.366388 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:49.369307 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:49.373913 (+ 4606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.373921 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.373937 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.374518 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.374926 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.374930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.374932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.375164 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.375820 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:49.375827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:49.376264 (+ 437us) client_negotiation.cc:769] Sending connection context
0923 17:24:49.376555 (+ 291us) client_negotiation.cc:240] Negotiation successful
0923 17:24:49.377024 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":435,"threads_started":1}
I20260923 17:24:49.380486 10117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:24:49 dist-test-slave-j0m4 krb5kdc[9963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.253@KRBTEST.COM for kudu/127.28.72.253@KRBTEST.COM
I20260923 17:24:49.396819 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.381318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:37403 (local address 127.0.0.1:53534)
0923 17:24:49.381561 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.381582 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:49.381706 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:49.382496 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:49.382504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:49.383041 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:49.383315 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.383329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.384796 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.384802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.385660 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.385672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.386519 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.386523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.386788 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.387694 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:49.387713 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:49.390147 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:49.393958 (+ 3811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.393963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.393967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.394369 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.394777 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.394784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.394787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.394945 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.395708 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:49.395713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:49.395857 (+ 144us) client_negotiation.cc:769] Sending connection context
0923 17:24:49.396012 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 17:24:49.396340 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":4}
I20260923 17:24:49.396854 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.381426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.28.72.253:37403)
0923 17:24:49.382062 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.382068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:49.382094 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:24:49.382139 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:49.382145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:49.382328 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:49.382443 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:49.383498 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.384648 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.385803 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.386366 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.386507 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.390449 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:49.390465 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:49.390479 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:49.390508 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:49.393771 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.394530 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.394537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.394540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.394667 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.395163 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.395168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.395171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.395586 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:49.395785 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:49.396167 (+ 382us) server_negotiation.cc:299] Negotiation successful
0923 17:24:49.396484 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":480,"thread_start_us":331,"threads_started":1}
I20260923 17:24:49.399374 10117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:24:49.400604 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.400069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37108)
0923 17:24:49.400267 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.400426 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260923 17:24:49.400885 10067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
I20260923 17:24:49.402793 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.402349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37124)
0923 17:24:49.402584 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.402673 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260923 17:24:49.403055 10117 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43677: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.252.keytab.
Entry for principal kudu/127.28.72.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.252.keytab.
I20260923 17:24:49.410472 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.252:43677
--webserver_interface=127.28.72.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.252.keytab
--principal=kudu/127.28.72.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:35607,127.28.72.253:37403,127.28.72.252:43677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:24:49.452445 10117 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } attempt: 1
I20260923 17:24:49.453759 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.453248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37138)
0923 17:24:49.453470 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.453603 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 17:24:49.454933 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.454565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37154)
0923 17:24:49.454740 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.454815 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:24:49.455150 10117 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43677: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
I20260923 17:24:49.523433 10117 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } attempt: 2
I20260923 17:24:49.524722 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.524179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37160)
0923 17:24:49.524470 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.524562 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 17:24:49.525894 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.525505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37166)
0923 17:24:49.525668 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.525757 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260923 17:24:49.526235 10117 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43677: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
I20260923 17:24:49.528087 10043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } attempt: 5
Sep 23 17:24:49 dist-test-slave-j0m4 krb5kdc[9963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184288, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for kudu/127.28.72.253@KRBTEST.COM
I20260923 17:24:49.544008 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.528735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:37403 (local address 127.0.0.1:53538)
0923 17:24:49.528939 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.528961 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:49.529099 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:49.529533 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:49.529540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:49.530025 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:49.530218 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.530228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.531708 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.531713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.532696 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.532710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.533705 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.533710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.533862 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.534578 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:49.534598 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:49.537082 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:49.541140 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.541145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.541148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.541594 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.542128 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.542132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.542135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.542289 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.543090 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:49.543094 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:49.543369 (+ 275us) client_negotiation.cc:769] Sending connection context
0923 17:24:49.543513 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 17:24:49.543748 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 17:24:49.544356 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.528946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53538 (local address 127.28.72.253:37403)
0923 17:24:49.529144 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.529149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:49.529170 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:24:49.529233 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:49.529238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:49.529314 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:49.529425 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:49.530404 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.531552 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.532862 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.533539 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.533677 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.537326 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:49.537334 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:49.537338 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:49.537381 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:49.540931 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.541757 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.541766 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.541771 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.541966 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.542426 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.542434 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.542438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.542917 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:49.543101 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:49.543690 (+ 589us) server_negotiation.cc:299] Negotiation successful
0923 17:24:49.543995 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:24:49.545643 10043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:24:49.546802 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.546373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37182)
0923 17:24:49.546558 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.546642 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 17:24:49.548008 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.547593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37184)
0923 17:24:49.547757 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.547874 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 17:24:49.548269 10043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43677: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
I20260923 17:24:49.599550 10043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } attempt: 1
I20260923 17:24:49.601122 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.600365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37198)
0923 17:24:49.600700 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.600849 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260923 17:24:49.602396 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.601965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37210)
0923 17:24:49.602135 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.602252 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 17:24:49.602680 10043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43677: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
I20260923 17:24:49.617518 10117 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } attempt: 3
I20260923 17:24:49.618872 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.618325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37224)
0923 17:24:49.618612 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.618708 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260923 17:24:49.620088 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.619642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37238)
0923 17:24:49.619889 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.619967 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260923 17:24:49.620316 10117 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43677: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
I20260923 17:24:49.671016 10043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } attempt: 2
I20260923 17:24:49.672498 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.671804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37248)
0923 17:24:49.672125 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.672263 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 17:24:49.673734 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.673287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37256)
0923 17:24:49.673462 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.673566 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 17:24:49.674014 10043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43677: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
I20260923 17:24:49.763568 10117 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } attempt: 4
I20260923 17:24:49.765017 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.764392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37270)
0923 17:24:49.764670 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.764791 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 17:24:49.766297 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.765829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37286)
0923 17:24:49.766034 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.766144 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 17:24:49.766677 10117 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43677: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
W20260923 17:24:49.775849 10124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:49.776244 10124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:49.776412 10124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 17:24:49.780254 10043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } attempt: 3
I20260923 17:24:49.781590 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.781008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37290)
0923 17:24:49.781280 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.781396 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 17:24:49.782739 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.782343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37300)
0923 17:24:49.782521 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.782613 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 17:24:49.783027 10043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43677: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43677: connect: Connection refused (error 111)
W20260923 17:24:49.785063 10124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:49.785195 10124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:49.785292 10124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:49.785388 10124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:49.785477 10124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:49.796358 10124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:35607,127.28.72.253:37403,127.28.72.252:43677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.252.keytab
--principal=kudu/127.28.72.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.252:43677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.72.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:49.797799 10124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:49.799271 10124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:49.812770 10132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:49.812726 10129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:49.812680 10130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:49.813812 10124 server_base.cc:1034] running on GCE node
I20260923 17:24:49.814852 10124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:49.816573 10124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:49.817896 10124 hybrid_clock.cc:648] HybridClock initialized: now 1790184289817816 us; error 78 us; skew 500 ppm
I20260923 17:24:49.818305 10124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:49 dist-test-slave-j0m4 krb5kdc[9963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:49.822072 10124 init.cc:377] Logged in from keytab as kudu/127.28.72.252@KRBTEST.COM (short username kudu)
I20260923 17:24:49.824060 10124 webserver.cc:460] Webserver started at http://127.28.72.252:33079/ using document root <none> and password file <none>
I20260923 17:24:49.824972 10124 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:49.825120 10124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:49.825469 10124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:49.828997 10124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/data/instance:
uuid: "f6b5e482ceda425986c579dfd0a459d3"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-j0m4"
server_key: "de5e3eefd9e1f24cc35ee08755c8d258"
server_key_iv: "887c558aabf8fa5fab680b52685626ba"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.829810 10124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/wal/instance:
uuid: "f6b5e482ceda425986c579dfd0a459d3"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-j0m4"
server_key: "de5e3eefd9e1f24cc35ee08755c8d258"
server_key_iv: "887c558aabf8fa5fab680b52685626ba"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.835503 10124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:24:49.839308 10139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:49.840945 10124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:24:49.841123 10124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/wal
uuid: "f6b5e482ceda425986c579dfd0a459d3"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-j0m4"
server_key: "de5e3eefd9e1f24cc35ee08755c8d258"
server_key_iv: "887c558aabf8fa5fab680b52685626ba"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.841311 10124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:49.861056 10124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:49.866122 10124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:49.866567 10124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:49.886346 10124 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.252:43677
I20260923 17:24:49.886475 10190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.252:43677 every 8 connection(s)
I20260923 17:24:49.888751 10124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/data/info.pb
I20260923 17:24:49.894143 10191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:49.895460 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 10124
I20260923 17:24:49.895670 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184070511153-28963-0/minicluster-data/master-2/wal/instance
I20260923 17:24:49.896304 28963 external_mini_cluster.cc:1428] Setting key f47414c5f3cbd866e974caad7fe2f872
I20260923 17:24:49.904004 10191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:24:49 dist-test-slave-j0m4 krb5kdc[9963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.252@KRBTEST.COM
Sep 23 17:24:49 dist-test-slave-j0m4 krb5kdc[9963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.252@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:24:49.923270 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.898492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.28.72.252:43677)
0923 17:24:49.899419 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.899432 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:49.899480 (+ 48us) server_negotiation.cc:407] Connection header received
0923 17:24:49.901177 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:49.901202 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:49.901651 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:49.902122 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:49.903789 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.905182 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.906998 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.908235 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.908423 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.913765 (+ 5342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:49.913777 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:49.913793 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:49.913845 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:49.917750 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.918968 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.918978 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.918989 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.919232 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.919783 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.919787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.919789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.920559 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:49.920784 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:49.921281 (+ 497us) server_negotiation.cc:299] Negotiation successful
0923 17:24:49.921623 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":374,"threads_started":1}
I20260923 17:24:49.926142 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.906634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35607 (local address 127.0.0.1:51700)
0923 17:24:49.907781 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.907825 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:49.908029 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:49.908454 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:49.908460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:49.909095 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:49.909423 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.909438 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.911079 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.911083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.912616 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.912630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.913765 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.913773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.914027 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.914813 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:49.914837 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:49.917912 (+ 3075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:49.921920 (+ 4008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.921929 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.921936 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.922505 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.922934 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.922939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.922942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.923095 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.925004 (+ 1909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:49.925010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:49.925193 (+ 183us) client_negotiation.cc:769] Sending connection context
0923 17:24:49.925403 (+ 210us) client_negotiation.cc:240] Negotiation successful
0923 17:24:49.925767 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":797,"threads_started":1}
I20260923 17:24:49.926625 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.906692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.28.72.254:35607)
0923 17:24:49.907380 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.907387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:49.907975 (+ 588us) server_negotiation.cc:407] Connection header received
0923 17:24:49.908163 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:49.908170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:49.908250 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:49.908377 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:49.909605 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.910874 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.912783 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.913510 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.913617 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.918154 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:49.918162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:49.918168 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:49.918217 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:49.921798 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.922661 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.922665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.922667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.922818 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.923267 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.923272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.923274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.924849 (+ 1575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:49.925029 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:49.925563 (+ 534us) server_negotiation.cc:299] Negotiation successful
0923 17:24:49.925825 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":376,"threads_started":1}
I20260923 17:24:49.928203 10191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:24:49 dist-test-slave-j0m4 krb5kdc[9963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.252@KRBTEST.COM for kudu/127.28.72.253@KRBTEST.COM
Sep 23 17:24:49 dist-test-slave-j0m4 krb5kdc[9963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:24:49.943684 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.929094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:37403 (local address 127.0.0.1:53552)
0923 17:24:49.929253 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.929277 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:49.929434 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:49.929749 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:49.929754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:49.930212 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:49.930399 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.930410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.931487 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.931493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.932644 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.932658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.933547 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.933554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.933781 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.934432 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:49.934464 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:49.937226 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:49.940969 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.940973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.940976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.941388 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.941913 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.941919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.941924 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.942114 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.942738 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:49.942742 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:49.942947 (+ 205us) client_negotiation.cc:769] Sending connection context
0923 17:24:49.943147 (+ 200us) client_negotiation.cc:240] Negotiation successful
0923 17:24:49.943440 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 17:24:49.944018 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.929071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.28.72.253:37403)
0923 17:24:49.929284 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.929291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:49.929324 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:24:49.929554 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:49.929559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:49.929621 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:49.929695 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:49.930565 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.931351 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.932789 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.933358 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.933497 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.937427 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:49.937432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:49.937436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:49.937470 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:49.940808 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.941548 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.941555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.941559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.941753 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.942241 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.942244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.942247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.942554 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:49.942734 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:49.943327 (+ 593us) server_negotiation.cc:299] Negotiation successful
0923 17:24:49.943572 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:24:49.945281 10191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:24:49.945533 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.926820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.28.72.254:35607)
0923 17:24:49.927420 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.927427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:49.927447 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:24:49.927526 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:49.927532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:49.927627 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:49.927743 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:49.929449 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.930536 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.932171 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.932683 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.932836 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.938180 (+ 5344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:49.938189 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:49.938195 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:49.938249 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:49.941629 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.942649 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.942657 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.942664 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.942815 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.943291 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.943297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.943300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.943750 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:49.944046 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:49.944679 (+ 633us) server_negotiation.cc:299] Negotiation successful
0923 17:24:49.945171 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":395,"threads_started":1}
Sep 23 17:24:49 dist-test-slave-j0m4 krb5kdc[9963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.252@KRBTEST.COM for kudu/127.28.72.252@KRBTEST.COM
Sep 23 17:24:49 dist-test-slave-j0m4 krb5kdc[9963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.253@KRBTEST.COM
I20260923 17:24:49.959226 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.946026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37312)
0923 17:24:49.946234 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.946253 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:49.946359 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:49.946807 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:49.946812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:49.947327 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:49.947531 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.947544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.948607 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.948614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.949498 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.949509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.950203 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.950208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.950420 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.951037 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:49.951054 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:49.953451 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:49.956603 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.956609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.956613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.957028 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.957391 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.957396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.957402 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.957558 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.958074 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:49.958079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:49.958225 (+ 146us) client_negotiation.cc:769] Sending connection context
0923 17:24:49.958390 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 17:24:49.958741 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 17:24:49.959250 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.946250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.28.72.252:43677)
0923 17:24:49.946435 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.946440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:49.946460 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:24:49.946515 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:49.946521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:49.946604 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:49.946715 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:49.947704 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.948475 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.949639 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.950049 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.950187 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.953647 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:49.953653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:49.953658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:49.953697 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:49.956464 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.957151 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.957155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.957157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.957280 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.957667 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.957671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.957673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.957953 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:49.958111 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:49.958545 (+ 434us) server_negotiation.cc:299] Negotiation successful
0923 17:24:49.958891 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":35}
I20260923 17:24:49.960284 10043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } attempt: 4
I20260923 17:24:49.963897 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.947817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.28.72.253:37403)
0923 17:24:49.948020 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.948028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:49.948058 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:24:49.948112 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:49.948118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:49.948187 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:49.948290 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:49.949567 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.950347 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.951903 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.952547 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.952683 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.957061 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:49.957071 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:49.957077 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:49.957125 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:49.960538 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.961536 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.961541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.961543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.961688 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.962172 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.962179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.962185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.962557 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:49.962834 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:49.963285 (+ 451us) server_negotiation.cc:299] Negotiation successful
0923 17:24:49.963544 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 17:24:49.965907 10191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3: Bootstrap starting.
I20260923 17:24:49.969565 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 23 17:24:49 dist-test-slave-j0m4 krb5kdc[9963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184288, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for kudu/127.28.72.252@KRBTEST.COM
I20260923 17:24:49.970261 10191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:49.971978 10191 log.cc:824] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:49.976269 10191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3: No bootstrap required, opened a new log
I20260923 17:24:49.978449 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.960904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37314)
0923 17:24:49.961139 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.961155 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:49.961240 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:49.962015 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:49.962020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:49.962515 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:49.962729 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.962743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.964216 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.964222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.965707 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.965726 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.966792 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.966798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.967022 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.967796 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:49.967822 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:49.970631 (+ 2809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:49.975221 (+ 4590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.975231 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.975235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.975860 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.976405 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.976409 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.976412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.976590 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.977388 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:49.977393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:49.977579 (+ 186us) client_negotiation.cc:769] Sending connection context
0923 17:24:49.977727 (+ 148us) client_negotiation.cc:240] Negotiation successful
0923 17:24:49.978045 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260923 17:24:49.979993 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.960917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.28.72.252:43677)
0923 17:24:49.961680 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.961685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:49.961701 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:24:49.961748 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:49.961755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:49.961832 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:49.961934 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:49.962902 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.964036 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.965862 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.966619 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.966745 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.970889 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:49.970896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:49.970900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:49.970939 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:49.974946 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.976048 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.976053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.976056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.976230 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.976742 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.976747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.976750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.977209 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:49.979198 (+ 1989us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:49.979352 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 17:24:49.979615 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613}
I20260923 17:24:49.987221 10191 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } }
I20260923 17:24:49.987898 10043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b: Bootstrap starting.
I20260923 17:24:49.988178 10191 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:49.988365 10191 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b5e482ceda425986c579dfd0a459d3, State: Initialized, Role: FOLLOWER
I20260923 17:24:49.989240 10191 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } }
I20260923 17:24:49.994045 10043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:49.998442 10191 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:49.999670 10206 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } } }
I20260923 17:24:50.000896 10206 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:50.003799 10043 log.cc:824] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:50.006345 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.976931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.28.72.254:35607)
0923 17:24:49.977160 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.977169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:49.983800 (+ 6631us) server_negotiation.cc:407] Connection header received
0923 17:24:49.984088 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:49.984094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:49.984171 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:49.984250 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:49.987441 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.988586 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.991239 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.992220 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.992359 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.997577 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:49.997585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:49.997589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:49.997637 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:50.001538 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:50.002876 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:50.002887 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:50.002893 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:50.003148 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:50.003801 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:50.003809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:50.003816 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:50.004304 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:50.004625 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:50.005407 (+ 782us) server_negotiation.cc:299] Negotiation successful
0923 17:24:50.005771 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:24:50.018651 10043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b: No bootstrap required, opened a new log
I20260923 17:24:50.020008 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.976967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.28.72.253:37403)
0923 17:24:49.978212 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.978221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:49.988097 (+ 9876us) server_negotiation.cc:407] Connection header received
0923 17:24:49.988194 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:49.988203 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:49.988317 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:49.988445 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:49.997329 (+ 8884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.998856 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.004225 (+ 5369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.006431 (+ 2206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.006606 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:50.012939 (+ 6333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:50.012951 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:50.012958 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:50.013018 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:50.016181 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:50.017190 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:50.017196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:50.017199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:50.017393 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:50.017981 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:50.017986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:50.017989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:50.018396 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:50.018643 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:50.019372 (+ 729us) server_negotiation.cc:299] Negotiation successful
0923 17:24:50.019643 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1087}
I20260923 17:24:50.031315 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.987918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.28.72.252:43677)
0923 17:24:49.988087 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.988093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:50.007647 (+ 19554us) server_negotiation.cc:407] Connection header received
0923 17:24:50.007951 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:50.007960 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:50.008071 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:50.008159 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:50.012862 (+ 4703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.014084 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.016475 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.017210 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.017366 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:50.021771 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:50.021784 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:50.021790 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:50.021832 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:50.026244 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:50.027275 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:50.027281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:50.027284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:50.027444 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:50.027968 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:50.027976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:50.027982 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:50.028516 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:50.030141 (+ 1625us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:50.030336 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 17:24:50.030674 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:24:50.032614 10043 raft_consensus.cc:348] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } }
I20260923 17:24:50.033155 10043 raft_consensus.cc:374] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:50.033335 10043 raft_consensus.cc:729] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97f835816814b879d35f2c7707b2c1b, State: Initialized, Role: FOLLOWER
I20260923 17:24:50.034178 10043 consensus_queue.cc:260] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } }
I20260923 17:24:50.035482 10220 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } } }
I20260923 17:24:50.036116 10220 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:50.036892 10043 sys_catalog.cc:564] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:50.027511 10124 master_runner.cc:186] Error getting master registration for 127.28.72.254:35607: OK, instance_id { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" instance_seqno: 1790184288811655 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:24:50.041860 10124 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 17:24:50.044463 10218 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:24:50.044699 10218 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260923 17:24:50.049937 9997 tablet.cc:2369] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:24:50.058548 10231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:24:50.058843 10231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:24:50.065912 10117 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } attempt: 5
I20260923 17:24:50.057946 9976 master_runner.cc:186] Error getting master registration for 127.28.72.253:37403: OK, instance_id { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" instance_seqno: 1790184289332768 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:24:50.073205 9976 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 23 17:24:50 dist-test-slave-j0m4 krb5kdc[9963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.253@KRBTEST.COM for kudu/127.28.72.252@KRBTEST.COM
I20260923 17:24:50.087841 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:50.066563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43677 (local address 127.0.0.1:37336)
0923 17:24:50.066734 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:50.066758 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:50.066878 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:50.067685 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:50.067690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:50.068251 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:50.068478 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:50.068493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.072126 (+ 3633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.072132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:50.073676 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:50.073695 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.074841 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.074848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:50.075144 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:50.075906 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:50.075934 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:50.079035 (+ 3101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:50.084597 (+ 5562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:50.084603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:50.084608 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:50.085231 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:50.085791 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:50.085797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:50.085801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:50.085979 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:50.086679 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:50.086685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:50.086871 (+ 186us) client_negotiation.cc:769] Sending connection context
0923 17:24:50.087051 (+ 180us) client_negotiation.cc:240] Negotiation successful
0923 17:24:50.087449 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 17:24:50.087944 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:50.066948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.28.72.252:43677)
0923 17:24:50.067143 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:50.067148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:50.067166 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:24:50.067228 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:50.067233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:50.067340 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:50.067446 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:50.068666 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.069868 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.073878 (+ 4010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.074641 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.074801 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:50.079319 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:50.079327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:50.079333 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:50.079393 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:50.084397 (+ 5004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:50.085397 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:50.085404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:50.085408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:50.085610 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:50.086121 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:50.086128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:50.086132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:50.086526 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:50.086692 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:50.087215 (+ 523us) server_negotiation.cc:299] Negotiation successful
0923 17:24:50.087500 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:24:50.094595 10117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46: Bootstrap starting.
I20260923 17:24:50.100099 10117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:50.101538 10117 log.cc:824] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:50.105362 10117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46: No bootstrap required, opened a new log
I20260923 17:24:50.112931 10206 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:50.113296 10206 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } }
I20260923 17:24:50.115151 10206 leader_election.cc:288] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97f835816814b879d35f2c7707b2c1b (127.28.72.254:35607), 8270a990a5454e82a94cf5fdd6da8e46 (127.28.72.253:37403)
W20260923 17:24:50.119199 10145 tablet.cc:2369] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:50.119065 10018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6b5e482ceda425986c579dfd0a459d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97f835816814b879d35f2c7707b2c1b" is_pre_election: true
I20260923 17:24:50.120110 10117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } }
I20260923 17:24:50.120807 10117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:50.120798 10018 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b5e482ceda425986c579dfd0a459d3 in term 0.
I20260923 17:24:50.121007 10117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8270a990a5454e82a94cf5fdd6da8e46, State: Initialized, Role: FOLLOWER
I20260923 17:24:50.121754 10142 leader_election.cc:302] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97f835816814b879d35f2c7707b2c1b, f6b5e482ceda425986c579dfd0a459d3; no voters:
I20260923 17:24:50.122287 10206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:24:50.122447 10206 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:24:50.122210 10117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } }
I20260923 17:24:50.122594 10206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:50.126288 10206 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } }
I20260923 17:24:50.127944 10018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6b5e482ceda425986c579dfd0a459d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97f835816814b879d35f2c7707b2c1b"
I20260923 17:24:50.128196 10018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:50.133543 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6b5e482ceda425986c579dfd0a459d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8270a990a5454e82a94cf5fdd6da8e46" is_pre_election: true
I20260923 17:24:50.134173 10092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b5e482ceda425986c579dfd0a459d3 in term 0.
I20260923 17:24:50.134732 10117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:50.136380 10234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } } }
I20260923 17:24:50.136837 10234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:50.136857 10018 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b5e482ceda425986c579dfd0a459d3 in term 1.
I20260923 17:24:50.139223 10142 leader_election.cc:302] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97f835816814b879d35f2c7707b2c1b, f6b5e482ceda425986c579dfd0a459d3; no voters:
I20260923 17:24:50.143399 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6b5e482ceda425986c579dfd0a459d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8270a990a5454e82a94cf5fdd6da8e46"
I20260923 17:24:50.143659 10092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:50.147667 10092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b5e482ceda425986c579dfd0a459d3 in term 1.
I20260923 17:24:50.148617 10206 leader_election.cc:288] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [CANDIDATE]: Term 1 election: Requested vote from peers a97f835816814b879d35f2c7707b2c1b (127.28.72.254:35607), 8270a990a5454e82a94cf5fdd6da8e46 (127.28.72.253:37403)
I20260923 17:24:50.149122 10206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:50.149593 10206 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [term 1 LEADER]: Becoming Leader. State: Replica: f6b5e482ceda425986c579dfd0a459d3, State: Running, Role: LEADER
I20260923 17:24:50.150622 10206 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } }
I20260923 17:24:50.168825 10236 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6b5e482ceda425986c579dfd0a459d3. Latest consensus state: current_term: 1 leader_uuid: "f6b5e482ceda425986c579dfd0a459d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } } }
I20260923 17:24:50.169389 10236 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:50.176661 10243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:50.179322 10243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:50.200196 10092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [term 1 FOLLOWER]: Refusing update from remote peer f6b5e482ceda425986c579dfd0a459d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:50.202673 10249 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:24:50.205533 10236 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:50.206467 10018 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [term 1 FOLLOWER]: Refusing update from remote peer f6b5e482ceda425986c579dfd0a459d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:50.207332 10236 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:50.225545 10220 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6b5e482ceda425986c579dfd0a459d3. Latest consensus state: current_term: 1 leader_uuid: "f6b5e482ceda425986c579dfd0a459d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } } }
I20260923 17:24:50.225930 10220 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:50.227310 10247 mvcc.cc:205] Tried to move back new op lower bound from 7332594852645781504 to 7332594852486815744. Current Snapshot: MvccSnapshot[applied={T|T < 7332594852645781504}]
I20260923 17:24:50.233409 10252 mvcc.cc:205] Tried to move back new op lower bound from 7332594852645781504 to 7332594852486815744. Current Snapshot: MvccSnapshot[applied={T|T < 7332594852645781504}]
I20260923 17:24:50.237334 10220 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6b5e482ceda425986c579dfd0a459d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } } }
I20260923 17:24:50.237634 10220 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:50.245075 10206 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6b5e482ceda425986c579dfd0a459d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } } }
I20260923 17:24:50.245452 10206 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:50.246440 10243 catalog_manager.cc:1347] Generated new cluster ID: c5dad02dee5445be9e6c3356e97c8e5a
I20260923 17:24:50.246533 10243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:50.263610 10243 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:50.265985 10243 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:50.273010 10234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6b5e482ceda425986c579dfd0a459d3. Latest consensus state: current_term: 1 leader_uuid: "f6b5e482ceda425986c579dfd0a459d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } } }
I20260923 17:24:50.273317 10234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:50.279958 10236 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6b5e482ceda425986c579dfd0a459d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } } }
I20260923 17:24:50.280279 10236 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:50.284125 10234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6b5e482ceda425986c579dfd0a459d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97f835816814b879d35f2c7707b2c1b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35607 } } peers { permanent_uuid: "8270a990a5454e82a94cf5fdd6da8e46" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 37403 } } peers { permanent_uuid: "f6b5e482ceda425986c579dfd0a459d3" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43677 } } }
I20260923 17:24:50.284436 10234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:50.293578 10243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6b5e482ceda425986c579dfd0a459d3: Generated new TSK 0
I20260923 17:24:50.295080 10243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:24:51.061055 10231 catalog_manager.cc:1259] Loaded cluster ID: c5dad02dee5445be9e6c3356e97c8e5a
I20260923 17:24:51.061174 10231 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b: loading cluster ID for follower catalog manager: success
I20260923 17:24:51.063861 10231 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b: acquiring CA information for follower catalog manager: success
I20260923 17:24:51.065850 10231 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a97f835816814b879d35f2c7707b2c1b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:24:51.207473 10249 catalog_manager.cc:1259] Loaded cluster ID: c5dad02dee5445be9e6c3356e97c8e5a
I20260923 17:24:51.207643 10249 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46: loading cluster ID for follower catalog manager: success
I20260923 17:24:51.210577 10249 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46: acquiring CA information for follower catalog manager: success
I20260923 17:24:51.212949 10249 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8270a990a5454e82a94cf5fdd6da8e46: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:24:51.403371 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 9976
I20260923 17:24:51.404888 10038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:51.521059 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 9976
I20260923 17:24:51.552510 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 10050
I20260923 17:24:51.554087 10112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:51.681429 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 10050
I20260923 17:24:51.702790 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 10124
I20260923 17:24:51.704094 10186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:51.819978 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 10124
2026-09-23T17:24:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4868 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:24:51 dist-test-slave-j0m4 krb5kdc[10269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:24:51 dist-test-slave-j0m4 krb5kdc[10269](info): set up 2 sockets
Sep 23 17:24:51 dist-test-slave-j0m4 krb5kdc[10269](info): commencing operation
krb5kdc: starting...
W20260923 17:24:52.936106 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:24:53 dist-test-slave-j0m4 krb5kdc[10269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:24:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:24:53.191344 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:44325
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:37373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:53.532255 10282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:53.532653 10282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:53.532838 10282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:53.540629 10282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:53.540808 10282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:53.540923 10282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:53.541028 10282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:53.541119 10282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:53.551316 10282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:37373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:44325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:44325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:53.552733 10282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:53.554162 10282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:53.566542 10290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:53.566601 10288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:53.566643 10287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:53.567569 10282 server_base.cc:1034] running on GCE node
I20260923 17:24:53.568506 10282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:53.570066 10282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:53.571331 10282 hybrid_clock.cc:648] HybridClock initialized: now 1790184293571287 us; error 69 us; skew 500 ppm
I20260923 17:24:53.571740 10282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:53 dist-test-slave-j0m4 krb5kdc[10269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184293, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:53.575335 10282 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:24:53.577296 10282 webserver.cc:460] Webserver started at http://127.28.72.254:40271/ using document root <none> and password file <none>
I20260923 17:24:53.578166 10282 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:53.578316 10282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:53.578644 10282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:53.581916 10282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "942f3b1d48404c04bc9f565572e85334"
format_stamp: "Formatted at 2026-09-23 17:24:53 on dist-test-slave-j0m4"
server_key: "60257b1974e35377c40c19d314e295b8"
server_key_iv: "f00068cf56783ad8a8b8f41177f591df"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.582737 10282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "942f3b1d48404c04bc9f565572e85334"
format_stamp: "Formatted at 2026-09-23 17:24:53 on dist-test-slave-j0m4"
server_key: "60257b1974e35377c40c19d314e295b8"
server_key_iv: "f00068cf56783ad8a8b8f41177f591df"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.588382 10282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:24:53.592118 10297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:53.593566 10282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:24:53.593770 10282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "942f3b1d48404c04bc9f565572e85334"
format_stamp: "Formatted at 2026-09-23 17:24:53 on dist-test-slave-j0m4"
server_key: "60257b1974e35377c40c19d314e295b8"
server_key_iv: "f00068cf56783ad8a8b8f41177f591df"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.594005 10282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:53.613261 10282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:53.617962 10282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:53.618268 10282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:53.637869 10282 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:44325
I20260923 17:24:53.637900 10348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:44325 every 8 connection(s)
I20260923 17:24:53.639390 10282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:53.643580 10349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:53.645848 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 10282
I20260923 17:24:53.646093 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:24:53.646740 28963 external_mini_cluster.cc:1428] Setting key 4a0f51335ec9795dee2633f93ec8bf92
I20260923 17:24:53.657583 10349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334: Bootstrap starting.
I20260923 17:24:53.661417 10349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:53.662595 10349 log.cc:824] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:24:53 dist-test-slave-j0m4 krb5kdc[10269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:24:53.666332 10349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334: No bootstrap required, opened a new log
I20260923 17:24:53.674077 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:53.648807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.28.72.254:44325)
0923 17:24:53.649757 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:53.649774 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:53.649836 (+ 62us) server_negotiation.cc:407] Connection header received
0923 17:24:53.652310 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:53.652342 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:53.652921 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:53.653413 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:53.654890 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.656965 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.658638 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.659828 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.660030 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:53.664760 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:53.664772 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:53.664793 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:53.664836 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:53.668959 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:53.669854 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:53.669864 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:53.669874 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:53.670090 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:53.670501 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:53.670506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:53.670510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:53.671294 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:53.671589 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:53.672075 (+ 486us) server_negotiation.cc:299] Negotiation successful
0923 17:24:53.672598 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":355,"threads_started":1}
I20260923 17:24:53.677325 10349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942f3b1d48404c04bc9f565572e85334" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44325 } }
I20260923 17:24:53.677784 10349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:53.677922 10349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 942f3b1d48404c04bc9f565572e85334, State: Initialized, Role: FOLLOWER
I20260923 17:24:53.678748 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:24:53.678612 10349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942f3b1d48404c04bc9f565572e85334" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44325 } }
I20260923 17:24:53.678912 10349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:53.679051 10349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:53.679279 10349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:53.682471 10349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942f3b1d48404c04bc9f565572e85334" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44325 } }
I20260923 17:24:53.683022 10349 leader_election.cc:302] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 942f3b1d48404c04bc9f565572e85334; no voters:
I20260923 17:24:53.683687 10349 leader_election.cc:288] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:53.683811 10354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:53.684652 10354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [term 1 LEADER]: Becoming Leader. State: Replica: 942f3b1d48404c04bc9f565572e85334, State: Running, Role: LEADER
I20260923 17:24:53.685200 10354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942f3b1d48404c04bc9f565572e85334" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44325 } }
I20260923 17:24:53.686043 10349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:53.688153 10355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "942f3b1d48404c04bc9f565572e85334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942f3b1d48404c04bc9f565572e85334" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44325 } } }
I20260923 17:24:53.688127 10356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 942f3b1d48404c04bc9f565572e85334. Latest consensus state: current_term: 1 leader_uuid: "942f3b1d48404c04bc9f565572e85334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942f3b1d48404c04bc9f565572e85334" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 44325 } } }
I20260923 17:24:53.688607 10355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:53.688772 10356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:53.690342 10359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:53.696094 10359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:53.705742 10359 catalog_manager.cc:1347] Generated new cluster ID: e033f0f6720b4c3d91926f23288b8b01
I20260923 17:24:53.705884 10359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:53.719008 10359 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:53.720968 10359 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:53.727569 10359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 942f3b1d48404c04bc9f565572e85334: Generated new TSK 0
I20260923 17:24:53.728615 10359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:24:53.741155 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 10282
I20260923 17:24:53.742559 10344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:53.838897 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 10282
2026-09-23T17:24:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2013 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:24:53 dist-test-slave-j0m4 krb5kdc[10377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:24:53 dist-test-slave-j0m4 krb5kdc[10377](info): set up 2 sockets
Sep 23 17:24:53 dist-test-slave-j0m4 krb5kdc[10377](info): commencing operation
krb5kdc: starting...
W20260923 17:24:54.959921 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:24:55 dist-test-slave-j0m4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:24:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:24:55.212380 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:40189
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:40189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:55.553787 10390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:55.554121 10390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:55.554214 10390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:55.561710 10390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:55.561847 10390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:55.561903 10390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:55.561956 10390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:55.562002 10390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:55.572187 10390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:40189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:40189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:55.573580 10390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:55.574929 10390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:55.586971 10396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:55.587255 10395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:55.586987 10398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:55.588152 10390 server_base.cc:1034] running on GCE node
I20260923 17:24:55.588949 10390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:55.590406 10390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:55.591616 10390 hybrid_clock.cc:648] HybridClock initialized: now 1790184295591571 us; error 30 us; skew 500 ppm
I20260923 17:24:55.591946 10390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:55 dist-test-slave-j0m4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:55.595252 10390 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:24:55.596935 10390 webserver.cc:460] Webserver started at http://127.28.72.254:41303/ using document root <none> and password file <none>
I20260923 17:24:55.597608 10390 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:55.597718 10390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:55.597986 10390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:55.601076 10390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "d2f4aacc0777445abd61ae0445865458"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-j0m4"
server_key: "a39dec4fb82f089a248ec3fbe3fdbd5e"
server_key_iv: "0a8bc5d107b6867d8a698ffafe72846d"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.601846 10390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "d2f4aacc0777445abd61ae0445865458"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-j0m4"
server_key: "a39dec4fb82f089a248ec3fbe3fdbd5e"
server_key_iv: "0a8bc5d107b6867d8a698ffafe72846d"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.607467 10390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:24:55.611294 10405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:55.613142 10390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:24:55.613329 10390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "d2f4aacc0777445abd61ae0445865458"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-j0m4"
server_key: "a39dec4fb82f089a248ec3fbe3fdbd5e"
server_key_iv: "0a8bc5d107b6867d8a698ffafe72846d"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.613559 10390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:55.635802 10390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:55.640771 10390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:55.641042 10390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:55.661801 10390 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:40189
I20260923 17:24:55.661831 10456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:40189 every 8 connection(s)
I20260923 17:24:55.663529 10390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:55.667688 10457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:55.668074 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 10390
I20260923 17:24:55.668314 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:24:55.669106 28963 external_mini_cluster.cc:1428] Setting key 89b7c665920522b00ea4e9d1c9d79774
I20260923 17:24:55.682755 10457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Bootstrap starting.
Sep 23 17:24:55 dist-test-slave-j0m4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:24:55.686110 10457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:55.687680 10457 log.cc:824] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:55.691901 10457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: No bootstrap required, opened a new log
I20260923 17:24:55.697234 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:55.671309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.28.72.254:40189)
0923 17:24:55.672120 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:55.672135 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:55.672186 (+ 51us) server_negotiation.cc:407] Connection header received
0923 17:24:55.674694 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:55.674727 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:55.675325 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:55.675822 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:55.677457 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.679575 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.681023 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.682098 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.682292 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:55.686641 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:55.686657 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:55.686686 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:55.686737 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:55.691682 (+ 4945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:55.692638 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:55.692651 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:55.692660 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:55.692886 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:55.693297 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:55.693305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:55.693309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:55.694286 (+ 977us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:55.694716 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:55.695136 (+ 420us) server_negotiation.cc:299] Negotiation successful
0923 17:24:55.695678 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":339,"threads_started":1}
I20260923 17:24:55.702455 10457 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } }
I20260923 17:24:55.702862 10457 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:55.702947 10457 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f4aacc0777445abd61ae0445865458, State: Initialized, Role: FOLLOWER
I20260923 17:24:55.703640 10457 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } }
I20260923 17:24:55.703912 10457 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:55.704124 10457 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:55.704326 10457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:55.706934 10457 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } }
I20260923 17:24:55.707619 10457 leader_election.cc:302] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2f4aacc0777445abd61ae0445865458; no voters:
I20260923 17:24:55.708343 10457 leader_election.cc:288] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:55.708464 10462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:55.710327 10457 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:55.711455 10462 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 1 LEADER]: Becoming Leader. State: Replica: d2f4aacc0777445abd61ae0445865458, State: Running, Role: LEADER
I20260923 17:24:55.712105 10462 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } }
I20260923 17:24:55.715655 10464 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2f4aacc0777445abd61ae0445865458. Latest consensus state: current_term: 1 leader_uuid: "d2f4aacc0777445abd61ae0445865458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } } }
I20260923 17:24:55.716140 10464 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:55.716243 10463 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2f4aacc0777445abd61ae0445865458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } } }
I20260923 17:24:55.716585 10463 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:55.717846 10477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:55.722694 10476 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:24:55.724354 10477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:55.733803 10477 catalog_manager.cc:1347] Generated new cluster ID: f7b85d22af9341f8a99194bf17b96765
I20260923 17:24:55.733942 10477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:55.773753 10477 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:55.775211 10477 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:55.782404 10477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Generated new TSK 0
I20260923 17:24:55.783361 10477 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:24:55.855943 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40189
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:56.228976 10483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:56.229302 10483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:56.229467 10483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:56.236984 10483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:56.237107 10483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:56.237224 10483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:24:56.247604 10483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:56.249002 10483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:56.250381 10483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:56.264552 10488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:56.264519 10491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:56.264416 10489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:56.265331 10483 server_base.cc:1034] running on GCE node
I20260923 17:24:56.265959 10483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:56.266938 10483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:56.268159 10483 hybrid_clock.cc:648] HybridClock initialized: now 1790184296268109 us; error 36 us; skew 500 ppm
I20260923 17:24:56.268560 10483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:56 dist-test-slave-j0m4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184296, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:56.272076 10483 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:24:56.273800 10483 webserver.cc:460] Webserver started at http://127.28.72.193:36223/ using document root <none> and password file <none>
I20260923 17:24:56.274659 10483 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:56.274821 10483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:56.275204 10483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:56.278756 10483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "74477d5e15c04057b17bee365a67dc29"
format_stamp: "Formatted at 2026-09-23 17:24:56 on dist-test-slave-j0m4"
server_key: "91e3a62b18072d33dcaa99ebf662f60f"
server_key_iv: "229973d250b778c0f194cc12da075828"
server_key_version: "encryptionkey@0"
I20260923 17:24:56.279717 10483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "74477d5e15c04057b17bee365a67dc29"
format_stamp: "Formatted at 2026-09-23 17:24:56 on dist-test-slave-j0m4"
server_key: "91e3a62b18072d33dcaa99ebf662f60f"
server_key_iv: "229973d250b778c0f194cc12da075828"
server_key_version: "encryptionkey@0"
I20260923 17:24:56.285985 10483 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 17:24:56.290025 10498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:56.291851 10483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:24:56.292068 10483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "74477d5e15c04057b17bee365a67dc29"
format_stamp: "Formatted at 2026-09-23 17:24:56 on dist-test-slave-j0m4"
server_key: "91e3a62b18072d33dcaa99ebf662f60f"
server_key_iv: "229973d250b778c0f194cc12da075828"
server_key_version: "encryptionkey@0"
I20260923 17:24:56.292330 10483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:56.314355 10483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:56.319257 10483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:56.319594 10483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:56.320657 10483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:56.322103 10483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:56.322220 10483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:56.322381 10483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:56.322459 10483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:56.358834 10483 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:43445
I20260923 17:24:56.358938 10610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:43445 every 8 connection(s)
I20260923 17:24:56.360281 10483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:24:56.367600 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 10483
I20260923 17:24:56.367851 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:24:56.368481 28963 external_mini_cluster.cc:1428] Setting key bbc98c01322d0719f680b3c1dc48dc25
Sep 23 17:24:56 dist-test-slave-j0m4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184296, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:24:56.384594 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:56.363521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:45061 (local address 127.28.72.254:40189)
0923 17:24:56.364062 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:56.364067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:56.365079 (+ 1012us) server_negotiation.cc:407] Connection header received
0923 17:24:56.367052 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:56.367058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:56.367180 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:56.367310 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:56.369199 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:56.370111 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:56.372500 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:56.372917 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:56.373029 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:56.377888 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:56.377893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:56.377896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:56.377935 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:56.381170 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:56.382025 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:56.382032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:56.382037 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:56.382171 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:56.382641 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:56.382646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:56.382649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:56.382978 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:56.383168 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:56.383965 (+ 797us) server_negotiation.cc:299] Negotiation successful
0923 17:24:56.384214 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":307,"threads_started":1}
I20260923 17:24:56.385711 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:56.364051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:45061)
0923 17:24:56.364889 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:56.364932 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:56.366731 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:56.367575 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:56.367585 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:56.368273 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:56.369007 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:56.369027 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:56.370289 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:56.370294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:56.372308 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:56.372327 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:56.373171 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:56.373181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:56.373552 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:56.374395 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:56.374436 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:56.377594 (+ 3158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:56.381352 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:56.381364 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:56.381383 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:56.381885 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:56.382299 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:56.382303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:56.382307 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:56.382518 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:56.383148 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:56.383157 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:56.383556 (+ 399us) client_negotiation.cc:769] Sending connection context
0923 17:24:56.383857 (+ 301us) client_negotiation.cc:240] Negotiation successful
0923 17:24:56.384472 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":312,"threads_started":1}
I20260923 17:24:56.387688 10611 heartbeater.cc:347] Connected to a master server at 127.28.72.254:40189
I20260923 17:24:56.388060 10611 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:56.389050 10611 heartbeater.cc:511] Master 127.28.72.254:40189 requested a full tablet report, sending...
I20260923 17:24:56.391855 10422 ts_manager.cc:194] Registered new tserver with Master: 74477d5e15c04057b17bee365a67dc29 (127.28.72.193:43445)
I20260923 17:24:56.394155 10422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:45061
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:24:56.430945 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40189
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:56.782280 10617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:56.782595 10617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:56.782764 10617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:56.790807 10617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:56.790933 10617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:56.791152 10617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:24:56.803076 10617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:56.804728 10617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:56.806245 10617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:56.822276 10623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:56.822276 10622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:56.822424 10625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:56.823360 10617 server_base.cc:1034] running on GCE node
I20260923 17:24:56.824052 10617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:56.825053 10617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:56.826287 10617 hybrid_clock.cc:648] HybridClock initialized: now 1790184296826234 us; error 42 us; skew 500 ppm
I20260923 17:24:56.826658 10617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:56 dist-test-slave-j0m4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184296, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:56.830061 10617 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:24:56.831719 10617 webserver.cc:460] Webserver started at http://127.28.72.194:33805/ using document root <none> and password file <none>
I20260923 17:24:56.832434 10617 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:56.832541 10617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:56.832798 10617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:56.836077 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "1978aa0b774e4c0ca886471786e05574"
format_stamp: "Formatted at 2026-09-23 17:24:56 on dist-test-slave-j0m4"
server_key: "2329a681b4b754fad1de160f12e947e5"
server_key_iv: "790ddbb469b788da22822df2ff67b7e2"
server_key_version: "encryptionkey@0"
I20260923 17:24:56.836944 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "1978aa0b774e4c0ca886471786e05574"
format_stamp: "Formatted at 2026-09-23 17:24:56 on dist-test-slave-j0m4"
server_key: "2329a681b4b754fad1de160f12e947e5"
server_key_iv: "790ddbb469b788da22822df2ff67b7e2"
server_key_version: "encryptionkey@0"
I20260923 17:24:56.842715 10617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:24:56.846771 10632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:56.848469 10617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:24:56.848629 10617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "1978aa0b774e4c0ca886471786e05574"
format_stamp: "Formatted at 2026-09-23 17:24:56 on dist-test-slave-j0m4"
server_key: "2329a681b4b754fad1de160f12e947e5"
server_key_iv: "790ddbb469b788da22822df2ff67b7e2"
server_key_version: "encryptionkey@0"
I20260923 17:24:56.848826 10617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:56.875350 10617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:56.880317 10617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:56.880604 10617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:56.881597 10617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:56.883009 10617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:56.883093 10617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:56.883253 10617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:56.883302 10617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:56.919790 10617 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:40677
I20260923 17:24:56.919902 10745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:40677 every 8 connection(s)
I20260923 17:24:56.921182 10617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:24:56.926512 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 10617
I20260923 17:24:56.926726 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:24:56.927384 28963 external_mini_cluster.cc:1428] Setting key 09038cab9e9d7ed0fbf43c2538c36dcf
Sep 23 17:24:56 dist-test-slave-j0m4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184296, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:24:56.949348 10748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:56.924203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:46275 (local address 127.28.72.254:40189)
0923 17:24:56.924760 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:56.924765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:56.926008 (+ 1243us) server_negotiation.cc:407] Connection header received
0923 17:24:56.928738 (+ 2730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:56.928744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:56.928835 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:56.928973 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:56.931468 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:56.932331 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:56.935246 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:56.935811 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:56.935934 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:56.940818 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:56.940827 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:56.940835 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:56.940897 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:56.944799 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:56.946008 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:56.946018 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:56.946023 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:56.946209 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:56.946850 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:56.946856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:56.946860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:56.947370 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:56.947585 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:56.948567 (+ 982us) server_negotiation.cc:299] Negotiation successful
0923 17:24:56.948894 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":311,"threads_started":1}
I20260923 17:24:56.950860 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:56.924970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:46275)
0923 17:24:56.925781 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:56.925826 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:56.928328 (+ 2502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:56.929245 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:56.929333 (+ 88us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:56.930178 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:56.931154 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:56.931190 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:56.932516 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:56.932524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:56.934951 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:56.934974 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:56.936098 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:56.936109 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:56.936490 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:56.937355 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:56.937392 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:56.940508 (+ 3116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:56.945025 (+ 4517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:56.945040 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:56.945064 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:56.945809 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:56.946366 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:56.946374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:56.946380 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:56.946666 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:56.947552 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:56.947565 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:56.948085 (+ 520us) client_negotiation.cc:769] Sending connection context
0923 17:24:56.948402 (+ 317us) client_negotiation.cc:240] Negotiation successful
0923 17:24:56.949056 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":306,"threads_started":1}
I20260923 17:24:56.952819 10746 heartbeater.cc:347] Connected to a master server at 127.28.72.254:40189
I20260923 17:24:56.953243 10746 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:56.954355 10746 heartbeater.cc:511] Master 127.28.72.254:40189 requested a full tablet report, sending...
I20260923 17:24:56.956272 10422 ts_manager.cc:194] Registered new tserver with Master: 1978aa0b774e4c0ca886471786e05574 (127.28.72.194:40677)
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
I20260923 17:24:56.957475 10422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:46275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:24:56.991262 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40189
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:57.337913 10752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:57.338295 10752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:57.338485 10752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:57.346319 10752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:57.346498 10752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:57.346738 10752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:24:57.357133 10752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:40189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:57.358654 10752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:57.360018 10752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:57.373721 10757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:57.373786 10758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:57.373749 10760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:57.374763 10752 server_base.cc:1034] running on GCE node
I20260923 17:24:57.375559 10752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:57.376646 10752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:57.377928 10752 hybrid_clock.cc:648] HybridClock initialized: now 1790184297377860 us; error 59 us; skew 500 ppm
I20260923 17:24:57.378295 10752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:57 dist-test-slave-j0m4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184297, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:57.381913 10752 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:24:57.383642 10752 webserver.cc:460] Webserver started at http://127.28.72.195:38325/ using document root <none> and password file <none>
I20260923 17:24:57.384413 10752 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:57.384572 10752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:57.384913 10752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:57.388192 10752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "adbb8fd650f0416a954f20f1310073ec"
format_stamp: "Formatted at 2026-09-23 17:24:57 on dist-test-slave-j0m4"
server_key: "f555abcadfcefcb3e1fba0e6c3ac038e"
server_key_iv: "e729458ecae6bda63f1446e5d93a8736"
server_key_version: "encryptionkey@0"
I20260923 17:24:57.389076 10752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "adbb8fd650f0416a954f20f1310073ec"
format_stamp: "Formatted at 2026-09-23 17:24:57 on dist-test-slave-j0m4"
server_key: "f555abcadfcefcb3e1fba0e6c3ac038e"
server_key_iv: "e729458ecae6bda63f1446e5d93a8736"
server_key_version: "encryptionkey@0"
I20260923 17:24:57.394621 10752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:24:57.398762 10611 heartbeater.cc:503] Master 127.28.72.254:40189 was elected leader, sending a full tablet report...
I20260923 17:24:57.398834 10767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:57.400981 10752 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:24:57.401144 10752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "adbb8fd650f0416a954f20f1310073ec"
format_stamp: "Formatted at 2026-09-23 17:24:57 on dist-test-slave-j0m4"
server_key: "f555abcadfcefcb3e1fba0e6c3ac038e"
server_key_iv: "e729458ecae6bda63f1446e5d93a8736"
server_key_version: "encryptionkey@0"
I20260923 17:24:57.401350 10752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:57.421382 10752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:57.426106 10752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:57.426365 10752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:57.427393 10752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:57.428954 10752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:57.429032 10752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:57.429124 10752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:57.429169 10752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:57.465140 10752 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:36739
I20260923 17:24:57.465178 10879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:36739 every 8 connection(s)
I20260923 17:24:57.466538 10752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:24:57.467264 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 10752
I20260923 17:24:57.467486 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:24:57.468125 28963 external_mini_cluster.cc:1428] Setting key df7f81e0f5e4d699cbd18acce98629a4
Sep 23 17:24:57 dist-test-slave-j0m4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184297, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:24:57.488708 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.469639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:41121 (local address 127.28.72.254:40189)
0923 17:24:57.472164 (+ 2525us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:57.472169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:57.472186 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:24:57.473681 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:57.473685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:57.473775 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:57.473882 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:57.475590 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.476367 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.478119 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.478526 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.478630 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.482414 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:57.482420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:57.482424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:57.482461 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:57.485550 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:57.486366 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:57.486372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:57.486375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:57.486509 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:57.486944 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:57.486947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:57.486949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:57.487317 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:57.487489 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:57.488241 (+ 752us) server_negotiation.cc:299] Negotiation successful
0923 17:24:57.488437 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2416,"thread_start_us":2321,"threads_started":1}
I20260923 17:24:57.489693 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.470242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:41121)
0923 17:24:57.471316 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:57.471379 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:57.473354 (+ 1975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:57.474129 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:57.474138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:57.474713 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:57.475420 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.475439 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.476528 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.476532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.477953 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.477966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.478739 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.478743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.478967 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.479723 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:57.479751 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:57.482114 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:57.485717 (+ 3603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:57.485725 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:57.485740 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:57.486240 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:57.486613 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:57.486617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:57.486619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:57.486840 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:57.487474 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:57.487481 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:57.487871 (+ 390us) client_negotiation.cc:769] Sending connection context
0923 17:24:57.488116 (+ 245us) client_negotiation.cc:240] Negotiation successful
0923 17:24:57.488544 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":385,"threads_started":1}
I20260923 17:24:57.491341 10880 heartbeater.cc:347] Connected to a master server at 127.28.72.254:40189
I20260923 17:24:57.491629 10880 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:57.492393 10880 heartbeater.cc:511] Master 127.28.72.254:40189 requested a full tablet report, sending...
I20260923 17:24:57.493894 10422 ts_manager.cc:194] Registered new tserver with Master: adbb8fd650f0416a954f20f1310073ec (127.28.72.195:36739)
I20260923 17:24:57.494737 10422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:41121
I20260923 17:24:57.497167 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:24:57.515054 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.501777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51050 (local address 127.28.72.254:40189)
0923 17:24:57.501948 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:57.501978 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:57.502426 (+ 448us) server_negotiation.cc:407] Connection header received
0923 17:24:57.502659 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:57.502669 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:57.502746 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:57.502856 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:57.504269 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.505070 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.506444 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.506839 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.506977 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.509458 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:57.509463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:57.509466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:57.509501 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:57.512181 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:57.512872 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:57.512884 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:57.512886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:57.513012 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:57.513469 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:57.513476 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:57.513481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:57.513837 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:57.514121 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:57.514472 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 17:24:57.514709 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:24:57.519834 10422 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:24:57.526989 10422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:24:57.553719 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.541295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.28.72.195:36739)
0923 17:24:57.543595 (+ 2300us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:57.543607 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:57.543652 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:24:57.543785 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:57.543796 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:57.544125 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:57.545189 (+ 1064us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:57.545504 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.547273 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.550741 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.552298 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.552446 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.552702 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:57.553075 (+ 373us) server_negotiation.cc:299] Negotiation successful
0923 17:24:57.553353 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2094,"thread_start_us":1928,"threads_started":1}
I20260923 17:24:57.553706 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.540821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36739 (local address 127.0.0.1:51118)
0923 17:24:57.542221 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:57.542268 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:57.542487 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:57.544455 (+ 1968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:57.544465 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:57.544504 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:57.544943 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.544962 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.547442 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.547448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.550585 (+ 3137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.550600 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.552518 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.552525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.552749 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.552812 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 17:24:57.552957 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 17:24:57.553190 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":100,"thread_start_us":2460,"threads_started":1}
I20260923 17:24:57.554359 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.542131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50126 (local address 127.28.72.194:40677)
0923 17:24:57.544908 (+ 2777us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:57.544919 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:57.545502 (+ 583us) server_negotiation.cc:407] Connection header received
0923 17:24:57.545652 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:57.545659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:57.545863 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:57.546053 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:57.546597 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.548036 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.550604 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.552567 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.552700 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.552882 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:57.553806 (+ 924us) server_negotiation.cc:299] Negotiation successful
0923 17:24:57.554010 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2583,"thread_start_us":2789,"threads_started":1}
I20260923 17:24:57.554711 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.541331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.28.72.193:43445)
0923 17:24:57.543238 (+ 1907us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:57.543251 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:57.544709 (+ 1458us) server_negotiation.cc:407] Connection header received
0923 17:24:57.545055 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:57.545063 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:57.545360 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:57.545543 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:57.546355 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.547938 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.551018 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.552683 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.552794 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.553182 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:57.554223 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0923 17:24:57.554363 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1675,"thread_start_us":1476,"threads_started":1}
I20260923 17:24:57.554780 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.542171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:40677 (local address 127.0.0.1:50126)
0923 17:24:57.545369 (+ 3198us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:57.545389 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:57.545513 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:57.546050 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:57.546055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:57.546082 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:57.546448 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.546457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.548212 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.548217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.550363 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.550445 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.553053 (+ 2608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.553061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.553234 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.553266 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:24:57.553377 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 17:24:57.553527 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3065,"mutex_wait_us":128,"thread_start_us":2134,"threads_started":1}
I20260923 17:24:57.555336 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.540821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:43445 (local address 127.0.0.1:56112)
0923 17:24:57.544593 (+ 3772us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:57.544613 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:57.544819 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:57.545698 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:57.545702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:57.545724 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:57.546116 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.546128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.548111 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.548116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.550817 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.550827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.553395 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.553401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.553582 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.553613 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:24:57.553728 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 17:24:57.553880 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3624,"mutex_wait_us":131,"thread_start_us":4202,"threads_started":1}
I20260923 17:24:57.558514 10546 tablet_service.cc:1467] Processing CreateTablet for tablet 14bace043f1b4638b80fe6e84e1a11ad (DEFAULT_TABLE table=test-table [id=775dd565ccb144879071cf09fa3e90ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:57.558493 10815 tablet_service.cc:1467] Processing CreateTablet for tablet 14bace043f1b4638b80fe6e84e1a11ad (DEFAULT_TABLE table=test-table [id=775dd565ccb144879071cf09fa3e90ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:57.559041 10681 tablet_service.cc:1467] Processing CreateTablet for tablet 14bace043f1b4638b80fe6e84e1a11ad (DEFAULT_TABLE table=test-table [id=775dd565ccb144879071cf09fa3e90ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:57.559959 10546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bace043f1b4638b80fe6e84e1a11ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:57.560024 10815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bace043f1b4638b80fe6e84e1a11ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:57.560343 10681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bace043f1b4638b80fe6e84e1a11ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:57.572022 10899 tablet_bootstrap.cc:492] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29: Bootstrap starting.
I20260923 17:24:57.575873 10899 tablet_bootstrap.cc:654] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:57.577029 10899 log.cc:824] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:57.580015 10899 tablet_bootstrap.cc:492] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29: No bootstrap required, opened a new log
I20260923 17:24:57.580292 10899 ts_tablet_manager.cc:1397] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 17:24:57.583387 10900 tablet_bootstrap.cc:492] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec: Bootstrap starting.
I20260923 17:24:57.584304 10901 tablet_bootstrap.cc:492] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574: Bootstrap starting.
I20260923 17:24:57.587661 10900 tablet_bootstrap.cc:654] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:57.588469 10901 tablet_bootstrap.cc:654] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:57.588768 10900 log.cc:824] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:57.589709 10901 log.cc:824] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:57.589735 10899 raft_consensus.cc:348] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } } peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } }
I20260923 17:24:57.590224 10899 raft_consensus.cc:374] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:57.590447 10899 raft_consensus.cc:729] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74477d5e15c04057b17bee365a67dc29, State: Initialized, Role: FOLLOWER
I20260923 17:24:57.591147 10899 consensus_queue.cc:260] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } } peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } }
I20260923 17:24:57.592096 10900 tablet_bootstrap.cc:492] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec: No bootstrap required, opened a new log
I20260923 17:24:57.592396 10900 ts_tablet_manager.cc:1397] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260923 17:24:57.593027 10899 ts_tablet_manager.cc:1428] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260923 17:24:57.594161 10901 tablet_bootstrap.cc:492] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574: No bootstrap required, opened a new log
I20260923 17:24:57.594537 10901 ts_tablet_manager.cc:1397] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260923 17:24:57.601152 10900 raft_consensus.cc:348] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } } peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } }
I20260923 17:24:57.601562 10900 raft_consensus.cc:374] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:57.601687 10900 raft_consensus.cc:729] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbb8fd650f0416a954f20f1310073ec, State: Initialized, Role: FOLLOWER
I20260923 17:24:57.602315 10900 consensus_queue.cc:260] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } } peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } }
I20260923 17:24:57.603477 10880 heartbeater.cc:503] Master 127.28.72.254:40189 was elected leader, sending a full tablet report...
I20260923 17:24:57.603718 10901 raft_consensus.cc:348] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } } peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } }
I20260923 17:24:57.604096 10901 raft_consensus.cc:374] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:57.604174 10900 ts_tablet_manager.cc:1428] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260923 17:24:57.604218 10901 raft_consensus.cc:729] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1978aa0b774e4c0ca886471786e05574, State: Initialized, Role: FOLLOWER
I20260923 17:24:57.604894 10901 consensus_queue.cc:260] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } } peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } }
I20260923 17:24:57.606046 10746 heartbeater.cc:503] Master 127.28.72.254:40189 was elected leader, sending a full tablet report...
I20260923 17:24:57.606887 10901 ts_tablet_manager.cc:1428] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574: Time spent starting tablet: real 0.012s user 0.008s sys 0.002s
W20260923 17:24:57.612996 10612 tablet.cc:2369] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:24:57.673367 10747 tablet.cc:2369] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:24:57.718493 10881 tablet.cc:2369] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:57.852114 10905 raft_consensus.cc:482] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:57.852591 10905 raft_consensus.cc:504] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } } peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } }
I20260923 17:24:57.854529 10905 leader_election.cc:288] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adbb8fd650f0416a954f20f1310073ec (127.28.72.195:36739), 1978aa0b774e4c0ca886471786e05574 (127.28.72.194:40677)
I20260923 17:24:57.856498 10906 raft_consensus.cc:482] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:57.856940 10906 raft_consensus.cc:504] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } } peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } }
I20260923 17:24:57.859200 10906 leader_election.cc:288] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74477d5e15c04057b17bee365a67dc29 (127.28.72.193:43445), 1978aa0b774e4c0ca886471786e05574 (127.28.72.194:40677)
I20260923 17:24:57.863750 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.855075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:35861 (local address 127.28.72.194:40677)
0923 17:24:57.855311 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:57.855319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:57.855990 (+ 671us) server_negotiation.cc:407] Connection header received
0923 17:24:57.856280 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:57.856286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:57.856372 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:57.856468 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:57.857213 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.858252 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.860925 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.862344 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.862534 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.862615 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:57.863253 (+ 638us) server_negotiation.cc:299] Negotiation successful
0923 17:24:57.863435 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:24:57.864923 10908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.854941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:40677 (local address 127.28.72.193:35861)
0923 17:24:57.855835 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:57.855855 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:57.856046 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:57.856498 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:57.856504 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:57.856543 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:57.857009 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.857020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.858409 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.858416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.860713 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.860728 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.862782 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.862789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.863001 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.863057 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 17:24:57.864340 (+ 1283us) client_negotiation.cc:240] Negotiation successful
0923 17:24:57.864519 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":594,"threads_started":1}
I20260923 17:24:57.866446 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.854961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:37231 (local address 127.28.72.195:36739)
0923 17:24:57.855201 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:57.855208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:57.856651 (+ 1443us) server_negotiation.cc:407] Connection header received
0923 17:24:57.856901 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:57.856906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:57.856996 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:57.857102 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:57.857755 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.858915 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.863204 (+ 4289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.864594 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.864740 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.864814 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:57.866010 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0923 17:24:57.866163 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:24:57.867354 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.855699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36739 (local address 127.28.72.193:37231)
0923 17:24:57.856545 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:57.856564 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:57.856727 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:57.857137 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:57.857146 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:57.857179 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:57.857569 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.857584 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.860394 (+ 2810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.860400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.862965 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.862978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.865597 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.865604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.865797 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.865834 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 17:24:57.866714 (+ 880us) client_negotiation.cc:240] Negotiation successful
0923 17:24:57.866888 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":119,"thread_start_us":425,"threads_started":1}
I20260923 17:24:57.867961 10701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bace043f1b4638b80fe6e84e1a11ad" candidate_uuid: "74477d5e15c04057b17bee365a67dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978aa0b774e4c0ca886471786e05574" is_pre_election: true
I20260923 17:24:57.868625 10701 raft_consensus.cc:2454] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74477d5e15c04057b17bee365a67dc29 in term 0.
I20260923 17:24:57.868546 10835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bace043f1b4638b80fe6e84e1a11ad" candidate_uuid: "74477d5e15c04057b17bee365a67dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbb8fd650f0416a954f20f1310073ec" is_pre_election: true
I20260923 17:24:57.869076 10835 raft_consensus.cc:2454] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74477d5e15c04057b17bee365a67dc29 in term 0.
I20260923 17:24:57.869562 10502 leader_election.cc:302] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1978aa0b774e4c0ca886471786e05574, 74477d5e15c04057b17bee365a67dc29; no voters:
I20260923 17:24:57.870345 10905 raft_consensus.cc:2781] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:24:57.870535 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.859406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:43445 (local address 127.28.72.195:44319)
0923 17:24:57.859624 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:57.859641 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:57.859758 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:57.860406 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:57.860414 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:57.860456 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:57.860771 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.860786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.862739 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.862747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.866780 (+ 4033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.866796 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.869654 (+ 2858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.869660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.869894 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.869935 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 17:24:57.870049 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 17:24:57.870198 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":1481}
I20260923 17:24:57.870517 10905 raft_consensus.cc:482] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:24:57.870630 10905 raft_consensus.cc:3037] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:57.871440 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.859807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:44319 (local address 127.28.72.193:43445)
0923 17:24:57.859999 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:57.860005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:57.860024 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:24:57.860082 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:57.860086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:57.860163 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:57.860260 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:57.860981 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.862526 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.867882 (+ 5356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.869279 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.869460 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.869547 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:57.870954 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0923 17:24:57.871151 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":66}
I20260923 17:24:57.874398 10905 raft_consensus.cc:504] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } } peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } }
I20260923 17:24:57.875478 10905 leader_election.cc:288] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [CANDIDATE]: Term 1 election: Requested vote from peers adbb8fd650f0416a954f20f1310073ec (127.28.72.195:36739), 1978aa0b774e4c0ca886471786e05574 (127.28.72.194:40677)
I20260923 17:24:57.875725 10835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bace043f1b4638b80fe6e84e1a11ad" candidate_uuid: "74477d5e15c04057b17bee365a67dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbb8fd650f0416a954f20f1310073ec"
I20260923 17:24:57.875989 10835 raft_consensus.cc:3037] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:57.876610 10701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bace043f1b4638b80fe6e84e1a11ad" candidate_uuid: "74477d5e15c04057b17bee365a67dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978aa0b774e4c0ca886471786e05574"
I20260923 17:24:57.876641 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bace043f1b4638b80fe6e84e1a11ad" candidate_uuid: "adbb8fd650f0416a954f20f1310073ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74477d5e15c04057b17bee365a67dc29" is_pre_election: true
I20260923 17:24:57.876889 10701 raft_consensus.cc:3037] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:57.877099 10566 raft_consensus.cc:2379] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate adbb8fd650f0416a954f20f1310073ec in current term 1: Already voted for candidate 74477d5e15c04057b17bee365a67dc29 in this term.
I20260923 17:24:57.879304 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.859594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:40677 (local address 127.28.72.195:46435)
0923 17:24:57.870771 (+ 11177us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:57.870792 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:57.871674 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:57.872237 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:57.872242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:57.872272 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:57.872480 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.872492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.874179 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.874188 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.876444 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.876464 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.878432 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.878438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.878647 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.878683 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:24:57.878779 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 17:24:57.878918 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11066,"thread_start_us":15365,"threads_started":1}
I20260923 17:24:57.879344 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.860116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:46435 (local address 127.28.72.194:40677)
0923 17:24:57.864292 (+ 4176us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:57.864299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:57.870869 (+ 6570us) server_negotiation.cc:407] Connection header received
0923 17:24:57.871871 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:57.871877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:57.871963 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:57.872032 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:57.872651 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.873974 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.876666 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.878200 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.878409 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.878498 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:57.878847 (+ 349us) server_negotiation.cc:299] Negotiation successful
0923 17:24:57.879009 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3775,"thread_start_us":14480,"threads_started":1}
I20260923 17:24:57.879487 10835 raft_consensus.cc:2454] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74477d5e15c04057b17bee365a67dc29 in term 1.
I20260923 17:24:57.880100 10499 leader_election.cc:302] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74477d5e15c04057b17bee365a67dc29, adbb8fd650f0416a954f20f1310073ec; no voters:
I20260923 17:24:57.880532 10905 raft_consensus.cc:2781] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:57.880528 10701 raft_consensus.cc:2454] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74477d5e15c04057b17bee365a67dc29 in term 1.
I20260923 17:24:57.880436 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bace043f1b4638b80fe6e84e1a11ad" candidate_uuid: "adbb8fd650f0416a954f20f1310073ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978aa0b774e4c0ca886471786e05574" is_pre_election: true
I20260923 17:24:57.880946 10700 raft_consensus.cc:2379] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate adbb8fd650f0416a954f20f1310073ec in current term 1: Already voted for candidate 74477d5e15c04057b17bee365a67dc29 in this term.
I20260923 17:24:57.881134 10905 raft_consensus.cc:686] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [term 1 LEADER]: Becoming Leader. State: Replica: 74477d5e15c04057b17bee365a67dc29, State: Running, Role: LEADER
I20260923 17:24:57.881718 10771 leader_election.cc:302] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: adbb8fd650f0416a954f20f1310073ec; no voters: 1978aa0b774e4c0ca886471786e05574, 74477d5e15c04057b17bee365a67dc29
I20260923 17:24:57.881728 10905 consensus_queue.cc:237] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } } peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } }
I20260923 17:24:57.882174 10906 raft_consensus.cc:2726] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 17:24:57.887161 10420 catalog_manager.cc:5495] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74477d5e15c04057b17bee365a67dc29 (127.28.72.193). New cstate: current_term: 1 leader_uuid: "74477d5e15c04057b17bee365a67dc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } health_report { overall_health: UNKNOWN } } }
I20260923 17:24:57.926138 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.919496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.28.72.193:43445)
0923 17:24:57.919695 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:57.919700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:57.919720 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:24:57.920031 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:57.920036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:57.920118 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:57.920246 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:57.920709 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.921525 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.923697 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.924172 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.924346 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.924894 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:57.925422 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:57.925554 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:57.925727 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 17:24:57.925803 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:24:57.934803 10835 raft_consensus.cc:1261] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [term 1 FOLLOWER]: Refusing update from remote peer 74477d5e15c04057b17bee365a67dc29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:57.934828 10700 raft_consensus.cc:1261] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574 [term 1 FOLLOWER]: Refusing update from remote peer 74477d5e15c04057b17bee365a67dc29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:57.935789 10912 consensus_queue.cc:1035] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:57.936168 10905 consensus_queue.cc:1035] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:57.948995 10915 mvcc.cc:205] Tried to move back new op lower bound from 7332594884330409984 to 7332594884128100352. Current Snapshot: MvccSnapshot[applied={T|T < 7332594884330409984}]
I20260923 17:24:57.955135 10916 mvcc.cc:205] Tried to move back new op lower bound from 7332594884330409984 to 7332594884128100352. Current Snapshot: MvccSnapshot[applied={T|T < 7332594884330409984}]
I20260923 17:24:57.965955 10422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:57.966266 10422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:57.970569 10422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Sending DeleteTablet for 3 replicas of tablet 14bace043f1b4638b80fe6e84e1a11ad
I20260923 17:24:57.972038 10546 tablet_service.cc:1514] Processing DeleteTablet for tablet 14bace043f1b4638b80fe6e84e1a11ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:57 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:56112
I20260923 17:24:57.972183 10681 tablet_service.cc:1514] Processing DeleteTablet for tablet 14bace043f1b4638b80fe6e84e1a11ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:57 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:50126
I20260923 17:24:57.972277 10815 tablet_service.cc:1514] Processing DeleteTablet for tablet 14bace043f1b4638b80fe6e84e1a11ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:57 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:51118
I20260923 17:24:57.973593 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 10483
I20260923 17:24:57.986224 10927 tablet_replica.cc:330] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec: stopping tablet replica
I20260923 17:24:57.987136 10927 raft_consensus.cc:2229] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:57.988034 10927 raft_consensus.cc:2258] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:57.990727 10927 ts_tablet_manager.cc:1905] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:57.992484 10926 tablet_replica.cc:330] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574: stopping tablet replica
I20260923 17:24:57.992599 10925 tablet_replica.cc:330] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29: stopping tablet replica
I20260923 17:24:57.993125 10925 raft_consensus.cc:2229] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:24:57.993220 10926 raft_consensus.cc:2229] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:57.993841 10925 raft_consensus.cc:2258] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:57.993909 10926 raft_consensus.cc:2258] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 23 17:24:57 dist-test-slave-j0m4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:24:57.998531 10926 ts_tablet_manager.cc:1905] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:57.999225 10925 ts_tablet_manager.cc:1905] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:57.999441 10927 ts_tablet_manager.cc:1918] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:24:57.999673 10927 log.cc:1196] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/14bace043f1b4638b80fe6e84e1a11ad
I20260923 17:24:58.000427 10927 ts_tablet_manager.cc:1939] T 14bace043f1b4638b80fe6e84e1a11ad P adbb8fd650f0416a954f20f1310073ec: Deleting consensus metadata
I20260923 17:24:58.002547 10406 catalog_manager.cc:4841] TS adbb8fd650f0416a954f20f1310073ec (127.28.72.195:36739): tablet 14bace043f1b4638b80fe6e84e1a11ad (table test-table [id=775dd565ccb144879071cf09fa3e90ac]) successfully deleted
I20260923 17:24:58.007718 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.975367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.28.72.193:43445)
0923 17:24:57.975650 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:57.975659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:57.983745 (+ 8086us) server_negotiation.cc:407] Connection header received
0923 17:24:57.984015 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:57.984024 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:57.984145 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:57.984298 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:57.986935 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.988529 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.991293 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.992158 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.992315 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.998196 (+ 5881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:57.998212 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:57.998230 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:57.998285 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:58.003580 (+ 5295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:58.004633 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:58.004643 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:58.004646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:58.004822 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:58.005412 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:58.005417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:58.005420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:58.005970 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:58.006308 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:58.007052 (+ 744us) server_negotiation.cc:299] Negotiation successful
0923 17:24:58.007342 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 17:24:58.008875 10925 ts_tablet_manager.cc:1918] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:24:58.009212 10925 log.cc:1196] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/14bace043f1b4638b80fe6e84e1a11ad
I20260923 17:24:58.009810 10606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:58.009936 10925 ts_tablet_manager.cc:1939] T 14bace043f1b4638b80fe6e84e1a11ad P 74477d5e15c04057b17bee365a67dc29: Deleting consensus metadata
I20260923 17:24:58.010491 10926 ts_tablet_manager.cc:1918] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:24:58.010710 10926 log.cc:1196] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/14bace043f1b4638b80fe6e84e1a11ad
I20260923 17:24:58.011446 10926 ts_tablet_manager.cc:1939] T 14bace043f1b4638b80fe6e84e1a11ad P 1978aa0b774e4c0ca886471786e05574: Deleting consensus metadata
I20260923 17:24:58.013587 10409 catalog_manager.cc:4841] TS 1978aa0b774e4c0ca886471786e05574 (127.28.72.194:40677): tablet 14bace043f1b4638b80fe6e84e1a11ad (table test-table [id=775dd565ccb144879071cf09fa3e90ac]) successfully deleted
I20260923 17:24:58.114615 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 10483
I20260923 17:24:58.114564 10408 catalog_manager.cc:4841] TS 74477d5e15c04057b17bee365a67dc29 (127.28.72.193:43445): tablet 14bace043f1b4638b80fe6e84e1a11ad (table test-table [id=775dd565ccb144879071cf09fa3e90ac]) successfully deleted
I20260923 17:24:58.133561 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 10617
Sep 23 17:24:58 dist-test-slave-j0m4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:24:58.150097 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.134387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.28.72.194:40677)
0923 17:24:58.134598 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:58.134607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:58.134692 (+ 85us) server_negotiation.cc:407] Connection header received
0923 17:24:58.134913 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:58.134920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:58.135020 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:58.135167 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:58.136522 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.137233 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.138564 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.138987 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.139166 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.143976 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:58.143995 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:58.144012 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:58.144060 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:58.147042 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:58.147911 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:58.147921 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:58.147945 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:58.148146 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:58.148658 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:58.148663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:58.148665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:58.149042 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:58.149269 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:58.149654 (+ 385us) server_negotiation.cc:299] Negotiation successful
0923 17:24:58.149832 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 17:24:58.150751 10741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:58.244240 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 10617
I20260923 17:24:58.261909 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 10752
Sep 23 17:24:58 dist-test-slave-j0m4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:24:58.279855 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.262797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.28.72.195:36739)
0923 17:24:58.263000 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:58.263005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:58.263227 (+ 222us) server_negotiation.cc:407] Connection header received
0923 17:24:58.263364 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:58.263372 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:58.263462 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:58.263558 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:58.264982 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.265821 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.267552 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.268041 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.268159 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.272999 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:58.273018 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:58.273036 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:58.273080 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:58.276535 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:58.277522 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:58.277531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:58.277533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:58.277707 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:58.278135 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:58.278139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:58.278141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:58.278556 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:58.278789 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:58.279258 (+ 469us) server_negotiation.cc:299] Negotiation successful
0923 17:24:58.279457 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 17:24:58.280665 10875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:58.373844 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 10752
I20260923 17:24:58.391206 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 10390
I20260923 17:24:58.392403 10452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:24:58.505646 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 10390
I20260923 17:24:58.527688 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:40189
--webserver_interface=127.28.72.254
--webserver_port=41303
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:40189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:58.883311 10933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:58.883651 10933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:58.883814 10933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:58.891721 10933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:58.891846 10933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:58.891942 10933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:58.892048 10933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:58.892108 10933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:58.903218 10933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:40189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:40189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=41303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:58.904625 10933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:58.906013 10933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:58.930203 10942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:58.930231 10940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:58.930259 10939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:58.933030 10933 server_base.cc:1034] running on GCE node
I20260923 17:24:58.934051 10933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:58.935930 10933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:58.937227 10933 hybrid_clock.cc:648] HybridClock initialized: now 1790184298937162 us; error 59 us; skew 500 ppm
I20260923 17:24:58.937698 10933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:58 dist-test-slave-j0m4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:58.946954 10933 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:24:58.949086 10933 webserver.cc:460] Webserver started at http://127.28.72.254:41303/ using document root <none> and password file <none>
I20260923 17:24:58.949978 10933 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:58.950162 10933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:58.957279 10933 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:24:58.960901 10949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:58.962785 10933 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260923 17:24:58.963069 10933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "d2f4aacc0777445abd61ae0445865458"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-j0m4"
server_key: "a39dec4fb82f089a248ec3fbe3fdbd5e"
server_key_iv: "0a8bc5d107b6867d8a698ffafe72846d"
server_key_version: "encryptionkey@0"
I20260923 17:24:58.964440 10933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:58.993736 10933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:58.999449 10933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:58.999778 10933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:59.021617 10933 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:40189
I20260923 17:24:59.021632 11000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:40189 every 8 connection(s)
I20260923 17:24:59.023361 10933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:59.024672 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 10933
I20260923 17:24:59.025516 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:43445
--local_ip_for_outbound_sockets=127.28.72.193
--tserver_master_addrs=127.28.72.254:40189
--webserver_port=36223
--webserver_interface=127.28.72.193
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:24:59.031490 11001 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:24:59.037205 11001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Bootstrap starting.
I20260923 17:24:59.066766 11001 log.cc:824] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:59.080523 11001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:24:59.081146 11001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Bootstrap complete.
I20260923 17:24:59.090742 11001 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } }
I20260923 17:24:59.091784 11001 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f4aacc0777445abd61ae0445865458, State: Initialized, Role: FOLLOWER
I20260923 17:24:59.092482 11001 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } }
I20260923 17:24:59.092743 11001 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:59.092875 11001 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:59.093075 11001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:24:59.097371 11001 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } }
I20260923 17:24:59.097992 11001 leader_election.cc:302] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2f4aacc0777445abd61ae0445865458; no voters:
I20260923 17:24:59.098708 11001 leader_election.cc:288] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:24:59.098860 11005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:24:59.101468 11001 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:59.103448 11005 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 2 LEADER]: Becoming Leader. State: Replica: d2f4aacc0777445abd61ae0445865458, State: Running, Role: LEADER
I20260923 17:24:59.104372 11005 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } }
I20260923 17:24:59.107779 11007 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2f4aacc0777445abd61ae0445865458. Latest consensus state: current_term: 2 leader_uuid: "d2f4aacc0777445abd61ae0445865458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } } }
I20260923 17:24:59.108243 11007 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:59.109032 11006 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d2f4aacc0777445abd61ae0445865458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } } }
I20260923 17:24:59.109925 11006 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:59.119410 11012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:59.131721 11012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:59.134632 11012 catalog_manager.cc:1259] Loaded cluster ID: f7b85d22af9341f8a99194bf17b96765
I20260923 17:24:59.134864 11012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:59.138801 11012 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 17:24:59.159129 11012 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 17:24:59.170224 11012 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:59.172387 11012 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Loaded TSK: 0
I20260923 17:24:59.174178 11012 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:24:59.424335 11003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:59.424645 11003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:59.424793 11003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:59.432453 11003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:59.432581 11003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:59.432757 11003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:24:59.443084 11003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:43445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=36223
--tserver_master_addrs=127.28.72.254:40189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:59.444628 11003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:59.446036 11003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:59.462726 11028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:59.464213 11029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:59.465126 11031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:59.466351 11003 server_base.cc:1034] running on GCE node
I20260923 17:24:59.467316 11003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:59.468604 11003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:59.469897 11003 hybrid_clock.cc:648] HybridClock initialized: now 1790184299469799 us; error 88 us; skew 500 ppm
I20260923 17:24:59.470299 11003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:59 dist-test-slave-j0m4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184299, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:59.474059 11003 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:24:59.476114 11003 webserver.cc:460] Webserver started at http://127.28.72.193:36223/ using document root <none> and password file <none>
I20260923 17:24:59.476909 11003 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:59.477110 11003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:59.484220 11003 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260923 17:24:59.487528 11038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:59.489193 11003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:24:59.489394 11003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "74477d5e15c04057b17bee365a67dc29"
format_stamp: "Formatted at 2026-09-23 17:24:56 on dist-test-slave-j0m4"
server_key: "91e3a62b18072d33dcaa99ebf662f60f"
server_key_iv: "229973d250b778c0f194cc12da075828"
server_key_version: "encryptionkey@0"
I20260923 17:24:59.490422 11003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:59.509917 11003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:59.514740 11003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:59.515049 11003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:59.516198 11003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:59.517684 11003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:59.517812 11003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:59.517973 11003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:59.518066 11003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:59.557693 11003 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:43445
I20260923 17:24:59.557772 11150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:43445 every 8 connection(s)
I20260923 17:24:59.559448 11003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:24:59.566121 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 11003
I20260923 17:24:59.566764 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:40677
--local_ip_for_outbound_sockets=127.28.72.194
--tserver_master_addrs=127.28.72.254:40189
--webserver_port=33805
--webserver_interface=127.28.72.194
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:24:59 dist-test-slave-j0m4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184299, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:24:59.587226 11154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.562883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:55365)
0923 17:24:59.563793 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:59.563841 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:59.565689 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:59.567610 (+ 1921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:59.567625 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:59.568470 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:59.569519 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:59.569551 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.571534 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.571542 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:59.573792 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:59.573823 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.575062 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.575070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:59.575379 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:59.576105 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:59.576171 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:59.578765 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:59.582573 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:59.582583 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:59.582599 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:59.583060 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:59.583586 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:59.583591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:59.583595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:59.583803 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:59.584598 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:59.584607 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:59.584959 (+ 352us) client_negotiation.cc:769] Sending connection context
0923 17:24:59.585247 (+ 288us) client_negotiation.cc:240] Negotiation successful
0923 17:24:59.585845 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":423,"threads_started":1}
I20260923 17:24:59.587226 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.562729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:55365 (local address 127.28.72.254:40189)
0923 17:24:59.563498 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:59.563511 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:59.563985 (+ 474us) server_negotiation.cc:407] Connection header received
0923 17:24:59.566515 (+ 2530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:59.566530 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:59.566951 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:59.567384 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:59.569820 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.571315 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.573995 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.574752 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.574995 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:59.579185 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:59.579196 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:59.579213 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:59.579255 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:59.582406 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:59.583258 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:59.583271 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:59.583277 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:59.583453 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:59.583934 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:59.583939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:59.583941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:59.584447 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:59.584658 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:59.585477 (+ 819us) server_negotiation.cc:299] Negotiation successful
0923 17:24:59.585860 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":320,"threads_started":1}
I20260923 17:24:59.590226 11151 heartbeater.cc:347] Connected to a master server at 127.28.72.254:40189
I20260923 17:24:59.590626 11151 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:59.591681 11151 heartbeater.cc:511] Master 127.28.72.254:40189 requested a full tablet report, sending...
I20260923 17:24:59.594200 10966 ts_manager.cc:194] Registered new tserver with Master: 74477d5e15c04057b17bee365a67dc29 (127.28.72.193:43445)
I20260923 17:24:59.595898 10966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:55365
W20260923 17:24:59.920885 11155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:59.921190 11155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:59.921340 11155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:59.928836 11155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:59.928967 11155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:59.929085 11155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:24:59.939214 11155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:40677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=33805
--tserver_master_addrs=127.28.72.254:40189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:24:59.940547 11155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:59.941931 11155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:59.956013 11160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:59.955938 11161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:59.955977 11163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:59.956808 11155 server_base.cc:1034] running on GCE node
I20260923 17:24:59.957512 11155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:59.958547 11155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:59.959795 11155 hybrid_clock.cc:648] HybridClock initialized: now 1790184299959741 us; error 49 us; skew 500 ppm
I20260923 17:24:59.960171 11155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:24:59 dist-test-slave-j0m4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184299, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:59.963733 11155 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:24:59.965265 11155 webserver.cc:460] Webserver started at http://127.28.72.194:33805/ using document root <none> and password file <none>
I20260923 17:24:59.965994 11155 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:59.966159 11155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:59.972214 11155 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:24:59.975459 11170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:59.977012 11155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:24:59.977216 11155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "1978aa0b774e4c0ca886471786e05574"
format_stamp: "Formatted at 2026-09-23 17:24:56 on dist-test-slave-j0m4"
server_key: "2329a681b4b754fad1de160f12e947e5"
server_key_iv: "790ddbb469b788da22822df2ff67b7e2"
server_key_version: "encryptionkey@0"
I20260923 17:24:59.978235 11155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:59.993141 11155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:59.997733 11155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:59.998035 11155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:59.999171 11155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:00.000744 11155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:00.000833 11155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:00.000948 11155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:00.001026 11155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:00.040346 11155 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:40677
I20260923 17:25:00.040458 11282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:40677 every 8 connection(s)
I20260923 17:25:00.041906 11155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:25:00.044278 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 11155
I20260923 17:25:00.045111 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:36739
--local_ip_for_outbound_sockets=127.28.72.195
--tserver_master_addrs=127.28.72.254:40189
--webserver_port=38325
--webserver_interface=127.28.72.195
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:25:00 dist-test-slave-j0m4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184299, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:25:00.065030 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.044822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:37223 (local address 127.28.72.254:40189)
0923 17:25:00.045015 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.045020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.046702 (+ 1682us) server_negotiation.cc:407] Connection header received
0923 17:25:00.048765 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.048771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.048906 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.049021 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:00.050680 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.051587 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.053715 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.054157 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.054272 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.058760 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:00.058765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:00.058768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:00.058805 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:00.061633 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:00.062566 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:00.062573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:00.062576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:00.062710 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:00.063198 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:00.063204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:00.063208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:00.063505 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:00.063662 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:00.064418 (+ 756us) server_negotiation.cc:299] Negotiation successful
0923 17:25:00.064669 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:25:00.066094 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.045654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:37223)
0923 17:25:00.046528 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:00.046569 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:00.048466 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:00.049290 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:00.049300 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:00.049873 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:00.050494 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.050512 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.051765 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.051769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.053523 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.053542 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.054419 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.054429 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.054779 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.055622 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:00.055657 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:00.058504 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:00.061793 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:00.061804 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:00.061824 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:00.062430 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:00.062825 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:00.062830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:00.062834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:00.063048 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:00.063634 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:00.063643 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:00.064032 (+ 389us) client_negotiation.cc:769] Sending connection context
0923 17:25:00.064305 (+ 273us) client_negotiation.cc:240] Negotiation successful
0923 17:25:00.064898 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":314,"threads_started":1}
I20260923 17:25:00.067749 11283 heartbeater.cc:347] Connected to a master server at 127.28.72.254:40189
I20260923 17:25:00.068071 11283 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:00.068841 11283 heartbeater.cc:511] Master 127.28.72.254:40189 requested a full tablet report, sending...
I20260923 17:25:00.070664 10966 ts_manager.cc:194] Registered new tserver with Master: 1978aa0b774e4c0ca886471786e05574 (127.28.72.194:40677)
I20260923 17:25:00.071574 10966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:37223
W20260923 17:25:00.392598 11286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:00.392953 11286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:00.393131 11286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:00.400956 11286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:00.401077 11286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:00.401190 11286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:25:00.411772 11286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:36739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=38325
--tserver_master_addrs=127.28.72.254:40189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:25:00.413084 11286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:00.414393 11286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:00.428143 11291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:00.428107 11292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:00.428166 11294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:00.429590 11286 server_base.cc:1034] running on GCE node
I20260923 17:25:00.430320 11286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:00.431453 11286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:00.432683 11286 hybrid_clock.cc:648] HybridClock initialized: now 1790184300432634 us; error 39 us; skew 500 ppm
I20260923 17:25:00.433043 11286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:25:00 dist-test-slave-j0m4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184300, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:00.436718 11286 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:25:00.438443 11286 webserver.cc:460] Webserver started at http://127.28.72.195:38325/ using document root <none> and password file <none>
I20260923 17:25:00.439278 11286 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:00.439435 11286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:00.445101 11286 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:25:00.448088 11301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:00.449635 11286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:25:00.449846 11286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "adbb8fd650f0416a954f20f1310073ec"
format_stamp: "Formatted at 2026-09-23 17:24:57 on dist-test-slave-j0m4"
server_key: "f555abcadfcefcb3e1fba0e6c3ac038e"
server_key_iv: "e729458ecae6bda63f1446e5d93a8736"
server_key_version: "encryptionkey@0"
I20260923 17:25:00.450883 11286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:00.465464 11286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:00.470788 11286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:00.471101 11286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:00.472245 11286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:00.473703 11286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:00.473809 11286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:00.473924 11286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:00.473999 11286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:00.509892 11286 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:36739
I20260923 17:25:00.509986 11413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:36739 every 8 connection(s)
I20260923 17:25:00.511512 11286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:25:00.521657 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 11286
Sep 23 17:25:00 dist-test-slave-j0m4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184300, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:25:00.533874 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.514510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:35871 (local address 127.28.72.254:40189)
0923 17:25:00.514716 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.514720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.515915 (+ 1195us) server_negotiation.cc:407] Connection header received
0923 17:25:00.517770 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.517776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.517874 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.517989 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:00.519662 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.520505 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.522597 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.523462 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.523607 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.527474 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:00.527479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:00.527482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:00.527515 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:00.530265 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:00.531029 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:00.531042 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:00.531044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:00.531225 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:00.531688 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:00.531693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:00.531697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:00.532028 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:00.532186 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:00.533363 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0923 17:25:00.533608 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:25:00.535282 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.514957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:35871)
0923 17:25:00.515721 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:00.515759 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:00.517488 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:00.518237 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:00.518245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:00.518777 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:00.519483 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.519502 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.520672 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.520683 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.522437 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.522455 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.523701 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.523710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.523980 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.524712 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:00.524737 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:00.527238 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:00.530413 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:00.530423 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:00.530442 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:00.530895 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:00.531341 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:00.531346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:00.531350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:00.531559 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:00.532208 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:00.532217 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:00.532730 (+ 513us) client_negotiation.cc:769] Sending connection context
0923 17:25:00.533102 (+ 372us) client_negotiation.cc:240] Negotiation successful
0923 17:25:00.533723 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":328,"threads_started":1}
I20260923 17:25:00.537137 11414 heartbeater.cc:347] Connected to a master server at 127.28.72.254:40189
I20260923 17:25:00.537598 11414 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:00.538456 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.522754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.28.72.254:40189)
0923 17:25:00.523699 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.523705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.523731 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:25:00.523783 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.523786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.523842 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.523956 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:00.525548 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.526285 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.527953 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.528622 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.528769 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.531450 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:00.531456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:00.531462 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:00.531497 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:00.535023 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:00.536151 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:00.536157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:00.536160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:00.536341 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:00.536871 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:00.536874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:00.536876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:00.537245 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:00.537499 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:00.537813 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 17:25:00.538068 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":658,"threads_started":1}
I20260923 17:25:00.538607 11414 heartbeater.cc:511] Master 127.28.72.254:40189 requested a full tablet report, sending...
I20260923 17:25:00.540160 10966 ts_manager.cc:194] Registered new tserver with Master: adbb8fd650f0416a954f20f1310073ec (127.28.72.195:36739)
I20260923 17:25:00.541083 10966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:35871
I20260923 17:25:00.541602 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:25:00.550678 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.542685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51072 (local address 127.28.72.254:40189)
0923 17:25:00.542879 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.542883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.545148 (+ 2265us) server_negotiation.cc:407] Connection header received
0923 17:25:00.545308 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.545311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.545518 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.545640 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:00.546014 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.546751 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.548362 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.548830 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.548931 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.549449 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:00.549983 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:00.550092 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:00.550232 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 17:25:00.550325 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:25:00.551714 10966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:25:00.559372 10966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:25:00.596447 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.572973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:43445 (local address 127.0.0.1:56146)
0923 17:25:00.586385 (+ 13412us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:00.586408 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:00.586559 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:00.587241 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:00.587246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:00.587277 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:00.587875 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.587887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.589827 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.589832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.592904 (+ 3072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.592919 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.595307 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.595314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.595536 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.595622 (+ 86us) client_negotiation.cc:769] Sending connection context
0923 17:25:00.595756 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 17:25:00.595967 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13221,"mutex_wait_us":220,"thread_start_us":13864,"threads_started":1}
I20260923 17:25:00.597066 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.574061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50148 (local address 127.28.72.194:40677)
0923 17:25:00.584239 (+ 10178us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.584252 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.585714 (+ 1462us) server_negotiation.cc:407] Connection header received
0923 17:25:00.585865 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.585877 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.586182 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.586384 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:00.587899 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.589795 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.593410 (+ 3615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.595095 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.595291 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.595474 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:00.596484 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0923 17:25:00.596695 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9963,"thread_start_us":9786,"threads_started":1}
I20260923 17:25:00.597324 11427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.572921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:40677 (local address 127.0.0.1:50148)
0923 17:25:00.585132 (+ 12211us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:00.585214 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:00.585519 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:00.586711 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:00.586719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:00.586752 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:00.587672 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.587688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.590041 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.590048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.593220 (+ 3172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.593234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.595635 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.595641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.595829 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.595857 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 17:25:00.595921 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 17:25:00.596037 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11787,"mutex_wait_us":234,"thread_start_us":13082,"threads_started":1}
I20260923 17:25:00.597635 11424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.572996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56146 (local address 127.28.72.193:43445)
0923 17:25:00.582776 (+ 9780us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.582790 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.586654 (+ 3864us) server_negotiation.cc:407] Connection header received
0923 17:25:00.586761 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.586768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.587026 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.587247 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:00.588081 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.589661 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.593085 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.594726 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.594875 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.595161 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:00.597097 (+ 1936us) server_negotiation.cc:299] Negotiation successful
0923 17:25:00.597297 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9528,"thread_start_us":8924,"threads_started":1}
I20260923 17:25:00.603708 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.574581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.28.72.195:36739)
0923 17:25:00.583664 (+ 9083us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.583677 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.587374 (+ 3697us) server_negotiation.cc:407] Connection header received
0923 17:25:00.587725 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.587734 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.587945 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.588167 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:00.588820 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.590367 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.592567 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.601962 (+ 9395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.602168 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.602433 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:00.603165 (+ 732us) server_negotiation.cc:299] Negotiation successful
0923 17:25:00.603341 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8836,"thread_start_us":8623,"threads_started":1}
I20260923 17:25:00.603603 11086 tablet_service.cc:1467] Processing CreateTablet for tablet a0591b774b454e7fb5898b43fe01421b (DEFAULT_TABLE table=test-table [id=7186f76da57345cfbcfd2e1812333dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:00.604573 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.573377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36739 (local address 127.0.0.1:51136)
0923 17:25:00.587247 (+ 13870us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:00.587264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:00.587504 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:00.588178 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:00.588183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:00.588211 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:00.588588 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.588601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.590528 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.590532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.592383 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.592397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.602615 (+ 10218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.602623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.602868 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.602910 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 17:25:00.603980 (+ 1070us) client_negotiation.cc:240] Negotiation successful
0923 17:25:00.604144 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13708,"mutex_wait_us":6,"thread_start_us":11145,"threads_started":1}
I20260923 17:25:00.604406 11151 heartbeater.cc:503] Master 127.28.72.254:40189 was elected leader, sending a full tablet report...
I20260923 17:25:00.605342 11086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0591b774b454e7fb5898b43fe01421b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:00.608362 11349 tablet_service.cc:1467] Processing CreateTablet for tablet a0591b774b454e7fb5898b43fe01421b (DEFAULT_TABLE table=test-table [id=7186f76da57345cfbcfd2e1812333dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:00.609663 11349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0591b774b454e7fb5898b43fe01421b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:00.618327 11430 tablet_bootstrap.cc:492] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29: Bootstrap starting.
I20260923 17:25:00.619567 11218 tablet_service.cc:1467] Processing CreateTablet for tablet a0591b774b454e7fb5898b43fe01421b (DEFAULT_TABLE table=test-table [id=7186f76da57345cfbcfd2e1812333dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:00.621052 11218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0591b774b454e7fb5898b43fe01421b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:00.622999 11430 tablet_bootstrap.cc:654] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:00.624471 11430 log.cc:824] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:00.625267 11431 tablet_bootstrap.cc:492] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec: Bootstrap starting.
I20260923 17:25:00.629256 11430 tablet_bootstrap.cc:492] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29: No bootstrap required, opened a new log
I20260923 17:25:00.629559 11430 ts_tablet_manager.cc:1397] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260923 17:25:00.629895 11431 tablet_bootstrap.cc:654] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:00.630981 11431 log.cc:824] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:00.634476 11431 tablet_bootstrap.cc:492] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec: No bootstrap required, opened a new log
I20260923 17:25:00.634796 11431 ts_tablet_manager.cc:1397] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260923 17:25:00.637332 11434 tablet_bootstrap.cc:492] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574: Bootstrap starting.
I20260923 17:25:00.640681 11430 raft_consensus.cc:348] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } } peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } }
I20260923 17:25:00.641217 11430 raft_consensus.cc:374] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:00.641319 11430 raft_consensus.cc:729] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74477d5e15c04057b17bee365a67dc29, State: Initialized, Role: FOLLOWER
I20260923 17:25:00.641728 11434 tablet_bootstrap.cc:654] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:00.642032 11430 consensus_queue.cc:260] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } } peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } }
I20260923 17:25:00.643771 11430 ts_tablet_manager.cc:1428] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260923 17:25:00.644495 11434 log.cc:824] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:00.645673 11431 raft_consensus.cc:348] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } } peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } }
I20260923 17:25:00.646077 11431 raft_consensus.cc:374] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:00.646193 11431 raft_consensus.cc:729] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbb8fd650f0416a954f20f1310073ec, State: Initialized, Role: FOLLOWER
I20260923 17:25:00.646942 11431 consensus_queue.cc:260] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } } peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } }
I20260923 17:25:00.648182 11434 tablet_bootstrap.cc:492] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574: No bootstrap required, opened a new log
I20260923 17:25:00.648531 11434 ts_tablet_manager.cc:1397] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260923 17:25:00.651859 11414 heartbeater.cc:503] Master 127.28.72.254:40189 was elected leader, sending a full tablet report...
I20260923 17:25:00.652623 11431 ts_tablet_manager.cc:1428] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260923 17:25:00.658017 11434 raft_consensus.cc:348] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } } peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } }
I20260923 17:25:00.658408 11434 raft_consensus.cc:374] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:00.658516 11434 raft_consensus.cc:729] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1978aa0b774e4c0ca886471786e05574, State: Initialized, Role: FOLLOWER
I20260923 17:25:00.659200 11434 consensus_queue.cc:260] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } } peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } }
I20260923 17:25:00.660463 11283 heartbeater.cc:503] Master 127.28.72.254:40189 was elected leader, sending a full tablet report...
I20260923 17:25:00.661010 11434 ts_tablet_manager.cc:1428] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260923 17:25:00.763305 11415 tablet.cc:2369] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:25:00.794137 11284 tablet.cc:2369] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:25:00.812289 11152 tablet.cc:2369] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:25:00.813221 11438 raft_consensus.cc:482] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:25:00.813552 11438 raft_consensus.cc:504] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } } peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } }
I20260923 17:25:00.815409 11438 leader_election.cc:288] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74477d5e15c04057b17bee365a67dc29 (127.28.72.193:43445), adbb8fd650f0416a954f20f1310073ec (127.28.72.195:36739)
I20260923 17:25:00.822609 11424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.815880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:57719 (local address 127.28.72.193:43445)
0923 17:25:00.816109 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.816117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.816864 (+ 747us) server_negotiation.cc:407] Connection header received
0923 17:25:00.817095 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.817099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.817159 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.817265 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:00.817829 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.818567 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.820719 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.821608 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.821804 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.821900 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:00.822212 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 17:25:00.822342 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:25:00.822788 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.815625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:43445 (local address 127.28.72.194:57719)
0923 17:25:00.816732 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:00.816748 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:00.816942 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:00.817281 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:00.817286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:00.817310 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:00.817677 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.817686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.818693 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.818701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.820572 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.820584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.821874 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.821880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.822037 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.822068 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:25:00.822171 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 17:25:00.822309 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"mutex_wait_us":22,"spinlock_wait_cycles":28032,"thread_start_us":780,"threads_started":1}
I20260923 17:25:00.822932 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.815888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:58553 (local address 127.28.72.195:36739)
0923 17:25:00.816132 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.816141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.816966 (+ 825us) server_negotiation.cc:407] Connection header received
0923 17:25:00.817179 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.817183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.817241 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.817429 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:00.818051 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.818762 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.820724 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.821621 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.821805 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.821897 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:00.822214 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 17:25:00.822362 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:25:00.823256 11439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.815864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36739 (local address 127.28.72.194:58553)
0923 17:25:00.816845 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:00.816861 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:00.817023 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:00.817440 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:00.817448 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:00.817494 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:00.817900 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.817909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.818913 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.818917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.820574 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.820586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.821907 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.821914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.822064 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.822085 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:25:00.822187 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 17:25:00.822328 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":18,"spinlock_wait_cycles":768,"thread_start_us":510,"threads_started":1}
I20260923 17:25:00.823926 11106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0591b774b454e7fb5898b43fe01421b" candidate_uuid: "1978aa0b774e4c0ca886471786e05574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74477d5e15c04057b17bee365a67dc29" is_pre_election: true
I20260923 17:25:00.824222 11369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0591b774b454e7fb5898b43fe01421b" candidate_uuid: "1978aa0b774e4c0ca886471786e05574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbb8fd650f0416a954f20f1310073ec" is_pre_election: true
I20260923 17:25:00.824582 11106 raft_consensus.cc:2454] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1978aa0b774e4c0ca886471786e05574 in term 0.
I20260923 17:25:00.824744 11369 raft_consensus.cc:2454] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1978aa0b774e4c0ca886471786e05574 in term 0.
I20260923 17:25:00.825376 11171 leader_election.cc:302] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1978aa0b774e4c0ca886471786e05574, adbb8fd650f0416a954f20f1310073ec; no voters:
I20260923 17:25:00.825855 11438 raft_consensus.cc:2781] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:25:00.826038 11438 raft_consensus.cc:482] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:25:00.826167 11438 raft_consensus.cc:3037] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:00.828558 11438 raft_consensus.cc:504] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } } peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } }
I20260923 17:25:00.829197 11438 leader_election.cc:288] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [CANDIDATE]: Term 1 election: Requested vote from peers 74477d5e15c04057b17bee365a67dc29 (127.28.72.193:43445), adbb8fd650f0416a954f20f1310073ec (127.28.72.195:36739)
I20260923 17:25:00.829811 11369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0591b774b454e7fb5898b43fe01421b" candidate_uuid: "1978aa0b774e4c0ca886471786e05574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbb8fd650f0416a954f20f1310073ec"
I20260923 17:25:00.829811 11106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0591b774b454e7fb5898b43fe01421b" candidate_uuid: "1978aa0b774e4c0ca886471786e05574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74477d5e15c04057b17bee365a67dc29"
I20260923 17:25:00.830050 11369 raft_consensus.cc:3037] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:00.830061 11106 raft_consensus.cc:3037] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:00.833463 11106 raft_consensus.cc:2454] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1978aa0b774e4c0ca886471786e05574 in term 1.
I20260923 17:25:00.833463 11369 raft_consensus.cc:2454] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1978aa0b774e4c0ca886471786e05574 in term 1.
I20260923 17:25:00.834048 11171 leader_election.cc:302] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1978aa0b774e4c0ca886471786e05574, adbb8fd650f0416a954f20f1310073ec; no voters:
I20260923 17:25:00.834481 11438 raft_consensus.cc:2781] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:00.834983 11438 raft_consensus.cc:686] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [term 1 LEADER]: Becoming Leader. State: Replica: 1978aa0b774e4c0ca886471786e05574, State: Running, Role: LEADER
I20260923 17:25:00.835592 11438 consensus_queue.cc:237] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } } peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } }
I20260923 17:25:00.840373 10966 catalog_manager.cc:5495] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1978aa0b774e4c0ca886471786e05574 (127.28.72.194). New cstate: current_term: 1 leader_uuid: "1978aa0b774e4c0ca886471786e05574" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1978aa0b774e4c0ca886471786e05574" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 40677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 } health_report { overall_health: UNKNOWN } } }
I20260923 17:25:00.862803 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.856914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50160 (local address 127.28.72.194:40677)
0923 17:25:00.857117 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.857122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.857141 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:25:00.857297 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.857301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.857452 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.857582 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:00.857960 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.858788 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.860380 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.860812 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.860920 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.861523 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:00.862106 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:00.862230 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:00.862471 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 17:25:00.862558 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:25:00.872573 11106 raft_consensus.cc:1261] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29 [term 1 FOLLOWER]: Refusing update from remote peer 1978aa0b774e4c0ca886471786e05574: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:00.872598 11369 raft_consensus.cc:1261] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec [term 1 FOLLOWER]: Refusing update from remote peer 1978aa0b774e4c0ca886471786e05574: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:00.873977 11441 consensus_queue.cc:1035] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74477d5e15c04057b17bee365a67dc29" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 43445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:00.874531 11438 consensus_queue.cc:1035] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adbb8fd650f0416a954f20f1310073ec" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:00.889633 11444 mvcc.cc:205] Tried to move back new op lower bound from 7332594896360390656 to 7332594896226160640. Current Snapshot: MvccSnapshot[applied={T|T < 7332594896360390656}]
I20260923 17:25:00.903151 10965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:00.903519 10965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:00.907660 10965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Sending DeleteTablet for 3 replicas of tablet a0591b774b454e7fb5898b43fe01421b
I20260923 17:25:00.909144 11086 tablet_service.cc:1514] Processing DeleteTablet for tablet a0591b774b454e7fb5898b43fe01421b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:00 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:56146
I20260923 17:25:00.909153 11218 tablet_service.cc:1514] Processing DeleteTablet for tablet a0591b774b454e7fb5898b43fe01421b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:00 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:50148
I20260923 17:25:00.909435 11349 tablet_service.cc:1514] Processing DeleteTablet for tablet a0591b774b454e7fb5898b43fe01421b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:00 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:51136
I20260923 17:25:00.910131 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 11003
I20260923 17:25:00.919394 11455 tablet_replica.cc:330] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec: stopping tablet replica
I20260923 17:25:00.920070 11455 raft_consensus.cc:2229] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:00.920740 11455 raft_consensus.cc:2258] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:00.923309 11455 ts_tablet_manager.cc:1905] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:00.923920 11453 tablet_replica.cc:330] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29: stopping tablet replica
I20260923 17:25:00.924463 11453 raft_consensus.cc:2229] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:00.925055 11453 raft_consensus.cc:2258] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:00.929075 11454 tablet_replica.cc:330] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574: stopping tablet replica
I20260923 17:25:00.929592 11454 raft_consensus.cc:2229] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:25:00.930383 11453 ts_tablet_manager.cc:1905] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:00.931166 11454 raft_consensus.cc:2258] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:00.937963 11454 ts_tablet_manager.cc:1905] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:00.938133 11453 ts_tablet_manager.cc:1918] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:00.938460 11453 log.cc:1196] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/a0591b774b454e7fb5898b43fe01421b
I20260923 17:25:00.939026 11453 ts_tablet_manager.cc:1939] T a0591b774b454e7fb5898b43fe01421b P 74477d5e15c04057b17bee365a67dc29: Deleting consensus metadata
I20260923 17:25:00.939258 11455 ts_tablet_manager.cc:1918] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:00.939445 11455 log.cc:1196] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/a0591b774b454e7fb5898b43fe01421b
I20260923 17:25:00.940013 11455 ts_tablet_manager.cc:1939] T a0591b774b454e7fb5898b43fe01421b P adbb8fd650f0416a954f20f1310073ec: Deleting consensus metadata
I20260923 17:25:00.941159 10952 catalog_manager.cc:4841] TS 74477d5e15c04057b17bee365a67dc29 (127.28.72.193:43445): tablet a0591b774b454e7fb5898b43fe01421b (table test-table [id=7186f76da57345cfbcfd2e1812333dd7]) successfully deleted
I20260923 17:25:00.941816 11424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.912308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.28.72.193:43445)
0923 17:25:00.912554 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.912563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.912588 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:25:00.912666 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.912674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.912776 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.912896 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:00.915569 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.917120 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.922034 (+ 4914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.923009 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.923207 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.929267 (+ 6060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:00.929283 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:00.929301 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:00.929370 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:00.933653 (+ 4283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:00.934823 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:00.934833 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:00.934838 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:00.935021 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:00.935553 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:00.935564 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:00.935569 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:00.936069 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:00.936428 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:00.937799 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0923 17:25:00.938061 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 17:25:00.941956 10950 catalog_manager.cc:4841] TS adbb8fd650f0416a954f20f1310073ec (127.28.72.195:36739): tablet a0591b774b454e7fb5898b43fe01421b (table test-table [id=7186f76da57345cfbcfd2e1812333dd7]) successfully deleted
I20260923 17:25:00.942607 11146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:00.946949 11454 ts_tablet_manager.cc:1918] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:00.947212 11454 log.cc:1196] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/a0591b774b454e7fb5898b43fe01421b
I20260923 17:25:00.947836 11454 ts_tablet_manager.cc:1939] T a0591b774b454e7fb5898b43fe01421b P 1978aa0b774e4c0ca886471786e05574: Deleting consensus metadata
I20260923 17:25:00.949898 10953 catalog_manager.cc:4841] TS 1978aa0b774e4c0ca886471786e05574 (127.28.72.194:40677): tablet a0591b774b454e7fb5898b43fe01421b (table test-table [id=7186f76da57345cfbcfd2e1812333dd7]) successfully deleted
I20260923 17:25:01.072765 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 11003
I20260923 17:25:01.098604 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 11155
I20260923 17:25:01.117345 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:01.099491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.28.72.194:40677)
0923 17:25:01.099664 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:01.099668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:01.099730 (+ 62us) server_negotiation.cc:407] Connection header received
0923 17:25:01.099935 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:01.099939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:01.100024 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:01.100148 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:01.102252 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:01.103503 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:01.105699 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:01.106438 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:01.106571 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:01.110403 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:01.110420 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:01.110437 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:01.110487 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:01.114042 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:01.115073 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:01.115081 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:01.115083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:01.115306 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:01.115753 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:01.115756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:01.115758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:01.116180 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:01.116433 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:01.116784 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 17:25:01.116966 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:25:01.118271 11278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:01.235639 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 11155
I20260923 17:25:01.255692 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 11286
I20260923 17:25:01.270956 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:01.256549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.28.72.195:36739)
0923 17:25:01.256744 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:01.256751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:01.256858 (+ 107us) server_negotiation.cc:407] Connection header received
0923 17:25:01.257053 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:01.257058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:01.257126 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:01.257218 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:01.258562 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:01.259449 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:01.260917 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:01.261319 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:01.261486 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:01.264270 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:01.264288 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:01.264305 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:01.264344 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:01.267356 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:01.268200 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:01.268216 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:01.268218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:01.268497 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:01.269056 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:01.269061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:01.269063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:01.269525 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:01.269839 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:01.270288 (+ 449us) server_negotiation.cc:299] Negotiation successful
0923 17:25:01.270634 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:25:01.271737 11409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:01.364622 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 11286
I20260923 17:25:01.381939 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 10933
I20260923 17:25:01.383335 10996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:01.509815 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 10933
I20260923 17:25:01.533542 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:40189
--webserver_interface=127.28.72.254
--webserver_port=41303
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:40189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:01.916546 11460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:01.916878 11460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:01.917032 11460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:01.924844 11460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:01.925000 11460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:01.925081 11460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:01.925177 11460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:01.925263 11460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:25:01.936028 11460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:40189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:40189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=41303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:25:01.937561 11460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:01.939056 11460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:01.952059 11465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:01.951978 11466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:01.952028 11468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:01.953073 11460 server_base.cc:1034] running on GCE node
I20260923 17:25:01.953975 11460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:01.955693 11460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:01.957001 11460 hybrid_clock.cc:648] HybridClock initialized: now 1790184301956952 us; error 68 us; skew 500 ppm
I20260923 17:25:01.957406 11460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:25:01 dist-test-slave-j0m4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184301, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:01.961162 11460 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:25:01.962915 11460 webserver.cc:460] Webserver started at http://127.28.72.254:41303/ using document root <none> and password file <none>
I20260923 17:25:01.963758 11460 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:01.963892 11460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:01.970265 11460 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:25:01.973950 11475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:01.976070 11460 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:25:01.976384 11460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "d2f4aacc0777445abd61ae0445865458"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-j0m4"
server_key: "a39dec4fb82f089a248ec3fbe3fdbd5e"
server_key_iv: "0a8bc5d107b6867d8a698ffafe72846d"
server_key_version: "encryptionkey@0"
I20260923 17:25:01.977869 11460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:01.998407 11460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:02.003348 11460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:02.003616 11460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:02.024077 11460 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:40189
I20260923 17:25:02.024088 11526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:40189 every 8 connection(s)
I20260923 17:25:02.025965 11460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:25:02.030244 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 11460
I20260923 17:25:02.031308 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:43445
--local_ip_for_outbound_sockets=127.28.72.193
--tserver_master_addrs=127.28.72.254:40189
--webserver_port=36223
--webserver_interface=127.28.72.193
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:25:02.035233 11527 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:25:02.040671 11527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Bootstrap starting.
I20260923 17:25:02.062822 11527 log.cc:824] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:02.081626 11527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:25:02.082338 11527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458: Bootstrap complete.
I20260923 17:25:02.092114 11527 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } }
I20260923 17:25:02.093084 11527 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f4aacc0777445abd61ae0445865458, State: Initialized, Role: FOLLOWER
I20260923 17:25:02.093809 11527 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } }
I20260923 17:25:02.094086 11527 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:25:02.094215 11527 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:25:02.094446 11527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 2 FOLLOWER]: Advancing to term 3
I20260923 17:25:02.098712 11527 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } }
I20260923 17:25:02.099375 11527 leader_election.cc:302] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2f4aacc0777445abd61ae0445865458; no voters:
I20260923 17:25:02.100131 11527 leader_election.cc:288] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 17:25:02.100342 11531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 17:25:02.101513 11531 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [term 3 LEADER]: Becoming Leader. State: Replica: d2f4aacc0777445abd61ae0445865458, State: Running, Role: LEADER
I20260923 17:25:02.102209 11531 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } }
I20260923 17:25:02.102674 11527 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:02.105085 11532 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d2f4aacc0777445abd61ae0445865458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } } }
I20260923 17:25:02.105564 11532 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:02.106401 11533 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2f4aacc0777445abd61ae0445865458. Latest consensus state: current_term: 3 leader_uuid: "d2f4aacc0777445abd61ae0445865458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f4aacc0777445abd61ae0445865458" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40189 } } }
I20260923 17:25:02.106786 11533 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f4aacc0777445abd61ae0445865458 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:02.116047 11536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:25:02.125731 11536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:25:02.127621 11536 catalog_manager.cc:1259] Loaded cluster ID: f7b85d22af9341f8a99194bf17b96765
I20260923 17:25:02.127863 11536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 17:25:02.130681 11536 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fedc7ccb9ad google::LogMessage::Fail() at ??:0
@ 0x7fedc7cce88a google::LogMessage::SendToLog() at ??:0
@ 0x7fedc7ccb4de google::LogMessage::Flush() at ??:0
@ 0x7fedc7ccca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7feddd6fe512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7feddd6fb976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7feddd783610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
I20260923 17:25:02.158905 11549 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7feddd783439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fedca917345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fedca91a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fedca91a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fedca8f5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fedcdbcf6db start_thread at ??:0
@ 0x7fedc6b6561f clone at ??:0
W20260923 17:25:02.457156 11529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:02.457459 11529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:02.457613 11529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:02.465230 11529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:02.465364 11529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:02.465543 11529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:25:02.475885 11529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:43445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=36223
--tserver_master_addrs=127.28.72.254:40189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:25:02.477311 11529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:02.478729 11529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:02.492568 11571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:02.492671 11568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:02.492642 11569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:02.493763 11529 server_base.cc:1034] running on GCE node
I20260923 17:25:02.494434 11529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:02.495582 11529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:02.496810 11529 hybrid_clock.cc:648] HybridClock initialized: now 1790184302496753 us; error 52 us; skew 500 ppm
I20260923 17:25:02.497153 11529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:25:02 dist-test-slave-j0m4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184302, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:02.500841 11529 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:25:02.502537 11529 webserver.cc:460] Webserver started at http://127.28.72.193:36223/ using document root <none> and password file <none>
I20260923 17:25:02.503336 11529 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:02.503455 11529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:02.509346 11529 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 17:25:02.512392 11578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:02.514040 11529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:25:02.514199 11529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "74477d5e15c04057b17bee365a67dc29"
format_stamp: "Formatted at 2026-09-23 17:24:56 on dist-test-slave-j0m4"
server_key: "91e3a62b18072d33dcaa99ebf662f60f"
server_key_iv: "229973d250b778c0f194cc12da075828"
server_key_version: "encryptionkey@0"
I20260923 17:25:02.515314 11529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:02.537757 11529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:02.542661 11529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:02.542915 11529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:02.543983 11529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:02.545370 11529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:02.545468 11529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:02.545555 11529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:02.545611 11529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:02.583184 11529 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:43445
I20260923 17:25:02.583287 11690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:43445 every 8 connection(s)
I20260923 17:25:02.584877 11529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:25:02.590054 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:02.588100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:39929)
0923 17:25:02.588901 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:02.589160 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":338,"threads_started":1}
W20260923 17:25:02.590898 11691 heartbeater.cc:644] Failed to heartbeat to 127.28.72.254:40189 (0 consecutive failures): Network error: Failed to ping master at 127.28.72.254:40189: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
I20260923 17:25:02.591480 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 11529
I20260923 17:25:02.591913 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:02.591584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:40867)
0923 17:25:02.591729 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:02.591807 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 17:25:02.592185 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:40677
--local_ip_for_outbound_sockets=127.28.72.194
--tserver_master_addrs=127.28.72.254:40189
--webserver_port=33805
--webserver_interface=127.28.72.194
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:25:02.592847 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:02.592529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:42527)
0923 17:25:02.592666 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:02.592743 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 17:25:02.593097 11691 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:25:02.943974 11694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:02.944271 11694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:02.944375 11694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:02.952420 11694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:02.952543 11694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:02.952661 11694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:25:02.962954 11694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:40677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=33805
--tserver_master_addrs=127.28.72.254:40189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:25:02.964387 11694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:02.965903 11694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:02.980252 11699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:02.980278 11700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:02.980512 11702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:02.981109 11694 server_base.cc:1034] running on GCE node
I20260923 17:25:02.981769 11694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:02.982831 11694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:02.984078 11694 hybrid_clock.cc:648] HybridClock initialized: now 1790184302983999 us; error 71 us; skew 500 ppm
I20260923 17:25:02.984450 11694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:25:02 dist-test-slave-j0m4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184302, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:02.988138 11694 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:25:02.989827 11694 webserver.cc:460] Webserver started at http://127.28.72.194:33805/ using document root <none> and password file <none>
I20260923 17:25:02.990558 11694 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:02.990708 11694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:02.996474 11694 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:25:02.999543 11709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:03.001035 11694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:25:03.001238 11694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "1978aa0b774e4c0ca886471786e05574"
format_stamp: "Formatted at 2026-09-23 17:24:56 on dist-test-slave-j0m4"
server_key: "2329a681b4b754fad1de160f12e947e5"
server_key_iv: "790ddbb469b788da22822df2ff67b7e2"
server_key_version: "encryptionkey@0"
I20260923 17:25:03.002300 11694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:03.032137 11694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:03.037158 11694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:03.037506 11694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:03.038689 11694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:03.040235 11694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:03.040369 11694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:03.040503 11694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:03.040591 11694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:03.077292 11694 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:40677
I20260923 17:25:03.077489 11821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:40677 every 8 connection(s)
I20260923 17:25:03.078867 11694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:25:03.084405 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:03.082270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:55887)
0923 17:25:03.083085 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:03.083417 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":319,"threads_started":1}
W20260923 17:25:03.085150 11822 heartbeater.cc:644] Failed to heartbeat to 127.28.72.254:40189 (0 consecutive failures): Network error: Failed to ping master at 127.28.72.254:40189: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
I20260923 17:25:03.086041 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:03.085689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:55057)
0923 17:25:03.085842 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:03.085933 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 17:25:03.086984 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:03.086611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:58419)
0923 17:25:03.086772 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:03.086863 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 17:25:03.087280 11822 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:25:03.087749 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 11694
I20260923 17:25:03.088359 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:36739
--local_ip_for_outbound_sockets=127.28.72.195
--tserver_master_addrs=127.28.72.254:40189
--webserver_port=38325
--webserver_interface=127.28.72.195
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:43371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:03.427963 11825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:03.428264 11825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:03.428361 11825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:03.435858 11825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:03.435989 11825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:03.436112 11825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:25:03.445957 11825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:43371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:36739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=38325
--tserver_master_addrs=127.28.72.254:40189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:25:03.447248 11825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:03.448521 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:03.462150 11830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:03.462049 11833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:03.462077 11831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:03.462934 11825 server_base.cc:1034] running on GCE node
I20260923 17:25:03.463590 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:03.464612 11825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:03.465880 11825 hybrid_clock.cc:648] HybridClock initialized: now 1790184303465817 us; error 49 us; skew 500 ppm
I20260923 17:25:03.466244 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:25:03 dist-test-slave-j0m4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184303, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:03.469601 11825 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:25:03.471294 11825 webserver.cc:460] Webserver started at http://127.28.72.195:38325/ using document root <none> and password file <none>
I20260923 17:25:03.471995 11825 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:03.472102 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:03.477968 11825 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:25:03.480885 11840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:03.482422 11825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:25:03.482599 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "adbb8fd650f0416a954f20f1310073ec"
format_stamp: "Formatted at 2026-09-23 17:24:57 on dist-test-slave-j0m4"
server_key: "f555abcadfcefcb3e1fba0e6c3ac038e"
server_key_iv: "e729458ecae6bda63f1446e5d93a8736"
server_key_version: "encryptionkey@0"
I20260923 17:25:03.483695 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:03.497372 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:03.501952 11825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:03.502229 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:03.503257 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:03.504611 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:03.504698 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:03.504808 11825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:03.504879 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:03.538981 11825 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:36739
I20260923 17:25:03.539070 11952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:36739 every 8 connection(s)
I20260923 17:25:03.540548 11825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:25:03.542822 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 11825
I20260923 17:25:03.545974 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:03.543907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:53095)
0923 17:25:03.544832 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:03.545093 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":340,"threads_started":1}
W20260923 17:25:03.546685 11953 heartbeater.cc:644] Failed to heartbeat to 127.28.72.254:40189 (0 consecutive failures): Network error: Failed to ping master at 127.28.72.254:40189: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
I20260923 17:25:03.547595 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:03.547276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:41691)
0923 17:25:03.547422 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:03.547492 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 17:25:03.548564 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:03.548204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:44007)
0923 17:25:03.548327 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:03.548458 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:25:03.548830 11953 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:25:03.594555 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:03.593714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:40753)
0923 17:25:03.594196 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:03.594306 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":243,"threads_started":1}
I20260923 17:25:04.089076 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:04.088027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:38423)
0923 17:25:04.088618 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:04.088751 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":267,"threads_started":1}
I20260923 17:25:04.550307 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:04.549464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:49605)
0923 17:25:04.549948 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:04.550078 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":224,"threads_started":1}
I20260923 17:25:04.596392 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:04.595611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:60357)
0923 17:25:04.596069 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:04.596157 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":219,"threads_started":1}
I20260923 17:25:05.090685 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.089992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:34097)
0923 17:25:05.090428 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.090517 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":213,"threads_started":1}
I20260923 17:25:05.552165 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.551402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:37341)
0923 17:25:05.551908 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.551996 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":254,"threads_started":1}
I20260923 17:25:05.598156 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.597412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:46007)
0923 17:25:05.597860 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.597970 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":223,"threads_started":1}
I20260923 17:25:06.092392 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.091652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:60397)
0923 17:25:06.092119 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.092207 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":254,"threads_started":1}
I20260923 17:25:06.554103 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.553244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:56779)
0923 17:25:06.553808 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.553914 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":252,"threads_started":1}
I20260923 17:25:06.600242 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.599440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:45293)
0923 17:25:06.599968 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.600055 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":210,"threads_started":1}
I20260923 17:25:07.094506 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:07.093594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:38571)
0923 17:25:07.094135 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:07.094259 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":293,"threads_started":1}
I20260923 17:25:07.556082 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:07.555277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:37213)
0923 17:25:07.555790 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:07.555881 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":251,"threads_started":1}
I20260923 17:25:07.602062 11969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:07.601298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:38949)
0923 17:25:07.601796 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:07.601883 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":264,"threads_started":1}
I20260923 17:25:08.096436 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.095623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:44945)
0923 17:25:08.096162 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.096253 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":280,"threads_started":1}
I20260923 17:25:08.558013 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.557146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:44997)
0923 17:25:08.557748 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.557832 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":352,"threads_started":1}
I20260923 17:25:08.603829 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.602931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:45187)
0923 17:25:08.603575 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.603667 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":444,"threads_started":1}
I20260923 17:25:09.098397 11973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:09.097518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:59885)
0923 17:25:09.098042 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:09.098151 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":269,"threads_started":1}
I20260923 17:25:09.559963 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:09.558998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:40815)
0923 17:25:09.559653 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:09.559752 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":428,"threads_started":1}
I20260923 17:25:09.605675 11975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:09.604836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:43495)
0923 17:25:09.605343 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:09.605447 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":215,"threads_started":1}
I20260923 17:25:10.100474 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.099570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:57687)
0923 17:25:10.100126 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:10.100218 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":231,"threads_started":1}
I20260923 17:25:10.562000 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.561018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:48149)
0923 17:25:10.561641 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:10.561757 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":319,"threads_started":1}
I20260923 17:25:10.607709 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.606767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:36279)
0923 17:25:10.607449 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:10.607545 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":376,"threads_started":1}
I20260923 17:25:11.102756 11979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.101645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:38737)
0923 17:25:11.102355 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:11.102492 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":357,"threads_started":1}
I20260923 17:25:11.563717 11980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.562922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:32919)
0923 17:25:11.563461 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:11.563549 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":300,"threads_started":1}
I20260923 17:25:11.609496 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.608646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:46313)
0923 17:25:11.609225 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:11.609307 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":315,"threads_started":1}
I20260923 17:25:12.104719 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:12.103753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:50523)
0923 17:25:12.104447 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:12.104538 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":363,"threads_started":1}
I20260923 17:25:12.565733 11983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:12.564769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:40233)
0923 17:25:12.565372 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:12.565490 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":267,"threads_started":1}
I20260923 17:25:12.611527 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:12.610576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:42257)
0923 17:25:12.611260 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:12.611348 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":426,"threads_started":1}
I20260923 17:25:13.106714 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:13.105840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:34637)
0923 17:25:13.106393 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:13.106526 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":283,"threads_started":1}
I20260923 17:25:13.567557 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:13.566755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:52453)
0923 17:25:13.567292 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:13.567385 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":267,"threads_started":1}
I20260923 17:25:13.613301 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:13.612520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:41241)
0923 17:25:13.613007 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:13.613108 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":239,"threads_started":1}
I20260923 17:25:14.108683 11988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:14.107887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:48153)
0923 17:25:14.108377 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:14.108482 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":247,"threads_started":1}
I20260923 17:25:14.569662 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:14.568803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:34833)
0923 17:25:14.569335 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:14.569457 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":269,"threads_started":1}
I20260923 17:25:14.615247 11990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:14.614359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:40223)
0923 17:25:14.614863 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:14.614987 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":262,"threads_started":1}
I20260923 17:25:15.110627 11991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:15.109899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:52517)
0923 17:25:15.110361 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:15.110465 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":231,"threads_started":1}
I20260923 17:25:15.571589 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:15.570793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:52697)
0923 17:25:15.571333 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:15.571417 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":261,"threads_started":1}
I20260923 17:25:15.616940 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:15.616176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:48897)
0923 17:25:15.616631 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:15.616726 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":206,"threads_started":1}
I20260923 17:25:16.112581 11994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.111679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:45055)
0923 17:25:16.112190 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:16.112298 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":253,"threads_started":1}
I20260923 17:25:16.573601 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.572678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:59997)
0923 17:25:16.573253 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:16.573374 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":281,"threads_started":1}
I20260923 17:25:16.618717 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.617948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:53613)
0923 17:25:16.618438 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:16.618537 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":247,"threads_started":1}
I20260923 17:25:17.114362 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:17.113616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:36471)
0923 17:25:17.114064 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:17.114156 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":225,"threads_started":1}
I20260923 17:25:17.575591 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:17.574769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:41237)
0923 17:25:17.575307 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:17.575409 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":266,"threads_started":1}
I20260923 17:25:17.621007 11999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:17.620159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.193:51671)
0923 17:25:17.620672 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:17.620827 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":293,"threads_started":1}
I20260923 17:25:18.116438 12000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:18.115603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.194:43035)
0923 17:25:18.116135 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:18.116253 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":226,"threads_started":1}
I20260923 17:25:18.550051 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 11529
I20260923 17:25:18.577689 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:18.576898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:40189 (local address 127.28.72.195:48673)
0923 17:25:18.577422 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:18.577514 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":294,"threads_started":1}
I20260923 17:25:18.580508 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:18.551400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.28.72.193:43445)
0923 17:25:18.552127 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:18.552159 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:18.552204 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:25:18.554105 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:18.554116 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:18.554567 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:18.554955 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:18.556482 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.557692 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.559041 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.559814 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.560028 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:18.573105 (+ 13077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:18.573116 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:18.573138 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:18.573198 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:18.576909 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:18.577710 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:18.577718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:18.577725 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:18.577981 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:18.578403 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:18.578410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:18.578417 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:18.578969 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:18.579331 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:18.579769 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 17:25:18.580231 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":387,"threads_started":1}
I20260923 17:25:18.581842 11686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:18.671316 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 11529
I20260923 17:25:18.688524 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 11694
I20260923 17:25:18.707690 12005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:18.689557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.28.72.194:40677)
0923 17:25:18.690124 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:18.690138 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:18.690175 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:25:18.691919 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:18.691930 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:18.692377 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:18.692741 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:18.694325 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.695650 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.697029 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.698011 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.698220 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:18.701024 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:18.701032 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:18.701050 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:18.701097 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:18.704114 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:18.704861 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:18.704874 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:18.704882 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:18.705058 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:18.705469 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:18.705475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:18.705478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:18.706042 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:18.706383 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:18.706853 (+ 470us) server_negotiation.cc:299] Negotiation successful
0923 17:25:18.707344 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":281,"threads_started":1}
I20260923 17:25:18.709154 11817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:18.796644 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 11694
I20260923 17:25:18.815649 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 11825
I20260923 17:25:18.834296 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:18.816779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.28.72.195:36739)
0923 17:25:18.817338 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:18.817353 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:18.817393 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:25:18.819045 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:18.819055 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:18.819570 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:18.819923 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:18.821399 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.822626 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.824107 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.824815 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.825029 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:18.827781 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:18.827790 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:18.827807 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:18.827852 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:18.830690 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:18.831529 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:18.831537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:18.831543 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:18.831715 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:18.832141 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:18.832145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:18.832147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:18.832716 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:18.833057 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:18.833508 (+ 451us) server_negotiation.cc:299] Negotiation successful
0923 17:25:18.833973 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":261,"threads_started":1}
I20260923 17:25:18.835991 11948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:18.919843 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 11825
2026-09-23T17:25:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25083 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:25:19 dist-test-slave-j0m4 krb5kdc[12011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:25:19 dist-test-slave-j0m4 krb5kdc[12011](info): set up 2 sockets
Sep 23 17:25:19 dist-test-slave-j0m4 krb5kdc[12011](info): commencing operation
krb5kdc: starting...
W20260923 17:25:20.031960 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:25:20 dist-test-slave-j0m4 krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:25:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:25:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:25:20.291926 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:46851
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:46851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:20.637715 12024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:20.638113 12024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:20.638278 12024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:20.646064 12024 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:25:20.646210 12024 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:25:20.646317 12024 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:25:20.646404 12024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:20.646505 12024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:20.646636 12024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:20.646723 12024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:20.646806 12024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:25:20.657271 12024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:46851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:46851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:25:20.658715 12024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:20.660163 12024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:20.672664 12032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:20.672730 12029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:20.672747 12030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:20.673687 12024 server_base.cc:1034] running on GCE node
I20260923 17:25:20.674589 12024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:20.676287 12024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:20.677552 12024 hybrid_clock.cc:648] HybridClock initialized: now 1790184320677492 us; error 47 us; skew 500 ppm
I20260923 17:25:20.677907 12024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:25:20 dist-test-slave-j0m4 krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:20.681300 12024 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:25:20.682927 12024 webserver.cc:460] Webserver started at http://127.28.72.254:34121/ using document root <none> and password file <none>
I20260923 17:25:20.683725 12024 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:20.683857 12024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:20.684181 12024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:20.687462 12024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "4b9f217d69754c9f94cafce2891ef076"
format_stamp: "Formatted at 2026-09-23 17:25:20 on dist-test-slave-j0m4"
server_key: "2f2b066c943275468cd21afad03932e2"
server_key_iv: "b76db5170747720f893057c6a727d816"
server_key_version: "encryptionkey@0"
I20260923 17:25:20.688275 12024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "4b9f217d69754c9f94cafce2891ef076"
format_stamp: "Formatted at 2026-09-23 17:25:20 on dist-test-slave-j0m4"
server_key: "2f2b066c943275468cd21afad03932e2"
server_key_iv: "b76db5170747720f893057c6a727d816"
server_key_version: "encryptionkey@0"
I20260923 17:25:20.693876 12024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:25:20.697836 12039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:20.699383 12024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:25:20.699604 12024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "4b9f217d69754c9f94cafce2891ef076"
format_stamp: "Formatted at 2026-09-23 17:25:20 on dist-test-slave-j0m4"
server_key: "2f2b066c943275468cd21afad03932e2"
server_key_iv: "b76db5170747720f893057c6a727d816"
server_key_version: "encryptionkey@0"
I20260923 17:25:20.699848 12024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:20.718730 12024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:20.722993 12024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:20.723318 12024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:20.743209 12024 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:46851
I20260923 17:25:20.743273 12090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:46851 every 8 connection(s)
I20260923 17:25:20.744632 12024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:25:20.747285 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 12024
I20260923 17:25:20.747501 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:25:20.748287 28963 external_mini_cluster.cc:1428] Setting key 05012c46be185f6ca6f830d0fa1318c8
I20260923 17:25:20.748836 12091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:20.764537 12091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Bootstrap starting.
Sep 23 17:25:20 dist-test-slave-j0m4 krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:25:20.768105 12091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:20.769273 12091 log.cc:824] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:20.772324 12091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: No bootstrap required, opened a new log
I20260923 17:25:20.773855 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:20.750519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.28.72.254:46851)
0923 17:25:20.751350 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:20.751361 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:20.751404 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:25:20.753660 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:20.753685 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:20.754120 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:20.754526 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:20.756485 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:20.757870 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:20.759875 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:20.760700 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:20.760838 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:20.766036 (+ 5198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:20.766049 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:20.766066 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:20.766113 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:20.769538 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:20.770328 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:20.770338 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:20.770345 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:20.770527 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:20.770961 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:20.770965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:20.770967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:20.771598 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:20.771932 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:20.772405 (+ 473us) server_negotiation.cc:299] Negotiation successful
0923 17:25:20.772854 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":372,"threads_started":1}
I20260923 17:25:20.780478 12091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } }
I20260923 17:25:20.781049 12091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:20.781183 12091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9f217d69754c9f94cafce2891ef076, State: Initialized, Role: FOLLOWER
I20260923 17:25:20.781991 12091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } }
I20260923 17:25:20.782322 12091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:25:20.782526 12091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:25:20.782734 12091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:20.785668 12091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } }
I20260923 17:25:20.786233 12091 leader_election.cc:302] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b9f217d69754c9f94cafce2891ef076; no voters:
I20260923 17:25:20.786865 12091 leader_election.cc:288] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:25:20.787078 12096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:20.788261 12096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 1 LEADER]: Becoming Leader. State: Replica: 4b9f217d69754c9f94cafce2891ef076, State: Running, Role: LEADER
I20260923 17:25:20.788926 12096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } }
I20260923 17:25:20.789256 12091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:20.791900 12098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b9f217d69754c9f94cafce2891ef076. Latest consensus state: current_term: 1 leader_uuid: "4b9f217d69754c9f94cafce2891ef076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } } }
I20260923 17:25:20.792282 12098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:20.792788 12097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b9f217d69754c9f94cafce2891ef076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } } }
I20260923 17:25:20.793259 12097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:20.799026 12103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:25:20.805124 12103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:25:20.815515 12103 catalog_manager.cc:1347] Generated new cluster ID: efb6904c87e249dc996d0839a97fec2c
I20260923 17:25:20.815660 12103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:25:20.839529 12103 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:25:20.841583 12103 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:25:20.853101 12103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Generated new TSK 0
I20260923 17:25:20.854331 12103 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:25:20.932420 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:46851
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:21.298249 12117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:21.298591 12117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:21.298750 12117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:21.306838 12117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:21.307008 12117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:21.307224 12117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:25:21.317667 12117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:25:21.319038 12117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:21.320469 12117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:21.334774 12122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:21.334725 12123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:21.334915 12125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:21.336009 12117 server_base.cc:1034] running on GCE node
I20260923 17:25:21.336699 12117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:21.337734 12117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:21.339030 12117 hybrid_clock.cc:648] HybridClock initialized: now 1790184321338958 us; error 66 us; skew 500 ppm
I20260923 17:25:21.339457 12117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:25:21 dist-test-slave-j0m4 krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184321, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:21.342999 12117 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:25:21.344722 12117 webserver.cc:460] Webserver started at http://127.28.72.193:44037/ using document root <none> and password file <none>
I20260923 17:25:21.345458 12117 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:21.345609 12117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:21.345927 12117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:21.349056 12117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "09f394a6ef0349e9b56fae2c8302c9de"
format_stamp: "Formatted at 2026-09-23 17:25:21 on dist-test-slave-j0m4"
server_key: "9ebaac2a4080c4e56b103b4876fe1f78"
server_key_iv: "cae692ef66ac42a81302654aea67a915"
server_key_version: "encryptionkey@0"
I20260923 17:25:21.349808 12117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "09f394a6ef0349e9b56fae2c8302c9de"
format_stamp: "Formatted at 2026-09-23 17:25:21 on dist-test-slave-j0m4"
server_key: "9ebaac2a4080c4e56b103b4876fe1f78"
server_key_iv: "cae692ef66ac42a81302654aea67a915"
server_key_version: "encryptionkey@0"
I20260923 17:25:21.355448 12117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:25:21.359275 12132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:21.360715 12117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:25:21.360898 12117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "09f394a6ef0349e9b56fae2c8302c9de"
format_stamp: "Formatted at 2026-09-23 17:25:21 on dist-test-slave-j0m4"
server_key: "9ebaac2a4080c4e56b103b4876fe1f78"
server_key_iv: "cae692ef66ac42a81302654aea67a915"
server_key_version: "encryptionkey@0"
I20260923 17:25:21.361125 12117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:21.394992 12117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:21.400656 12117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:21.400995 12117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:21.402048 12117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:21.403631 12117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:21.403765 12117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:21.403934 12117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:21.404018 12117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:21.444196 12117 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:39849
I20260923 17:25:21.444247 12244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:39849 every 8 connection(s)
I20260923 17:25:21.445569 12117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:25:21.450873 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 12117
I20260923 17:25:21.451089 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:25:21.451764 28963 external_mini_cluster.cc:1428] Setting key b49086006aaaeecf413a11625cd43552
Sep 23 17:25:21 dist-test-slave-j0m4 krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184321, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:25:21.468878 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:21.448538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:44775 (local address 127.28.72.254:46851)
0923 17:25:21.449045 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:21.449063 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:21.450252 (+ 1189us) server_negotiation.cc:407] Connection header received
0923 17:25:21.452372 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:21.452377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:21.452475 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:21.452593 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:21.454472 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.455266 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.456884 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.457307 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.457450 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:21.461478 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:21.461483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:21.461487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:21.461525 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:21.464835 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:21.466018 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:21.466023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:21.466025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:21.466171 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:21.466746 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:21.466751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:21.466753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:21.467074 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:21.467385 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:21.468399 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0923 17:25:21.468605 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":241,"threads_started":1}
I20260923 17:25:21.470801 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:21.449092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:44775)
0923 17:25:21.450051 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:21.450097 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:21.452078 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:21.452893 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:21.452905 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:21.453678 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:21.454289 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:21.454308 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.455429 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.455433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:21.456720 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:21.456732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.457574 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.457582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:21.457842 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:21.458596 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:21.458622 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:21.461145 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:21.465039 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:21.465053 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:21.465075 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:21.465812 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:21.466305 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:21.466313 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:21.466319 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:21.466596 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:21.467380 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:21.467390 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:21.467926 (+ 536us) client_negotiation.cc:769] Sending connection context
0923 17:25:21.468268 (+ 342us) client_negotiation.cc:240] Negotiation successful
0923 17:25:21.468967 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":324,"threads_started":1}
I20260923 17:25:21.472808 12245 heartbeater.cc:347] Connected to a master server at 127.28.72.254:46851
I20260923 17:25:21.473212 12245 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:21.474097 12245 heartbeater.cc:511] Master 127.28.72.254:46851 requested a full tablet report, sending...
I20260923 17:25:21.476750 12056 ts_manager.cc:194] Registered new tserver with Master: 09f394a6ef0349e9b56fae2c8302c9de (127.28.72.193:39849)
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
I20260923 17:25:21.478374 12056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:44775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:25:21.515044 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:46851
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:21.866978 12251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:21.867367 12251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:21.867538 12251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:21.875138 12251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:21.875309 12251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:21.875502 12251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:25:21.885725 12251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:25:21.887609 12251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:21.889533 12251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:21.905053 12256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:21.905133 12257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:21.905085 12259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:21.906658 12251 server_base.cc:1034] running on GCE node
I20260923 17:25:21.907469 12251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:21.908581 12251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:21.909809 12251 hybrid_clock.cc:648] HybridClock initialized: now 1790184321909716 us; error 81 us; skew 500 ppm
I20260923 17:25:21.910215 12251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:25:21 dist-test-slave-j0m4 krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184321, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:21.913676 12251 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:25:21.915488 12251 webserver.cc:460] Webserver started at http://127.28.72.194:33629/ using document root <none> and password file <none>
I20260923 17:25:21.916244 12251 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:21.916397 12251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:21.916728 12251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:21.920135 12251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "89d0fd247bf44f9fba3197863d854460"
format_stamp: "Formatted at 2026-09-23 17:25:21 on dist-test-slave-j0m4"
server_key: "48bac22c6e255f610380e9e158943c69"
server_key_iv: "6f9f31754f6c9089c6ea99987d3c2097"
server_key_version: "encryptionkey@0"
I20260923 17:25:21.921052 12251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "89d0fd247bf44f9fba3197863d854460"
format_stamp: "Formatted at 2026-09-23 17:25:21 on dist-test-slave-j0m4"
server_key: "48bac22c6e255f610380e9e158943c69"
server_key_iv: "6f9f31754f6c9089c6ea99987d3c2097"
server_key_version: "encryptionkey@0"
I20260923 17:25:21.927011 12251 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 17:25:21.930837 12266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:21.932576 12251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 17:25:21.932790 12251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "89d0fd247bf44f9fba3197863d854460"
format_stamp: "Formatted at 2026-09-23 17:25:21 on dist-test-slave-j0m4"
server_key: "48bac22c6e255f610380e9e158943c69"
server_key_iv: "6f9f31754f6c9089c6ea99987d3c2097"
server_key_version: "encryptionkey@0"
I20260923 17:25:21.933039 12251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:21.955919 12251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:21.960448 12251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:21.960768 12251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:21.961903 12251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:21.963449 12251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:21.963558 12251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:21.963708 12251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:21.963783 12251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:21.998936 12251 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:44777
I20260923 17:25:21.998989 12378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:44777 every 8 connection(s)
I20260923 17:25:22.000402 12251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:25:22.005076 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 12251
I20260923 17:25:22.005275 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:25:22.005740 28963 external_mini_cluster.cc:1428] Setting key 6290e806440f754b29aac3cb72be1643
Sep 23 17:25:22 dist-test-slave-j0m4 krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184321, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:25:22.022207 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.003046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:59157 (local address 127.28.72.254:46851)
0923 17:25:22.003580 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:22.003585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:22.005536 (+ 1951us) server_negotiation.cc:407] Connection header received
0923 17:25:22.007340 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:22.007345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:22.007441 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:22.007571 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:22.009221 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.009974 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.011656 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.012098 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.012237 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.015961 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:22.015966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:22.015970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:22.016004 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:22.018761 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:22.019764 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:22.019769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:22.019771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:22.019899 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:22.020417 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:22.020420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:22.020422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:22.020690 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:22.020808 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:22.021738 (+ 930us) server_negotiation.cc:299] Negotiation successful
0923 17:25:22.021911 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":289,"threads_started":1}
I20260923 17:25:22.023799 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.003628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:59157)
0923 17:25:22.004627 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:22.004681 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:22.007009 (+ 2328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:22.007834 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:22.007843 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:22.008424 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:22.009034 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.009054 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.010145 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.010149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.011495 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.011508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.012343 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.012350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.012600 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.013315 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:22.013344 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:22.015662 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:22.018914 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:22.018926 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:22.018947 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:22.019630 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:22.020012 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:22.020020 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:22.020026 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:22.020302 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:22.020817 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:22.020828 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:22.021312 (+ 484us) client_negotiation.cc:769] Sending connection context
0923 17:25:22.021653 (+ 341us) client_negotiation.cc:240] Negotiation successful
0923 17:25:22.022255 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":369,"threads_started":1}
I20260923 17:25:22.025496 12379 heartbeater.cc:347] Connected to a master server at 127.28.72.254:46851
I20260923 17:25:22.025864 12379 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:22.026818 12379 heartbeater.cc:511] Master 127.28.72.254:46851 requested a full tablet report, sending...
I20260923 17:25:22.028610 12056 ts_manager.cc:194] Registered new tserver with Master: 89d0fd247bf44f9fba3197863d854460 (127.28.72.194:44777)
I20260923 17:25:22.029480 12056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:59157
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:25:22.063627 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:46851
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:22.397850 12385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:22.398205 12385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:22.398392 12385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:22.406162 12385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:22.406325 12385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:22.406503 12385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:25:22.416393 12385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:25:22.417769 12385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:22.419157 12385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:22.432654 12390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:22.432567 12391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:22.432623 12393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:22.433472 12385 server_base.cc:1034] running on GCE node
I20260923 17:25:22.434091 12385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:22.435096 12385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:22.436384 12385 hybrid_clock.cc:648] HybridClock initialized: now 1790184322436339 us; error 52 us; skew 500 ppm
I20260923 17:25:22.436786 12385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:25:22 dist-test-slave-j0m4 krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184322, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:22.440131 12385 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:25:22.441591 12385 webserver.cc:460] Webserver started at http://127.28.72.195:36601/ using document root <none> and password file <none>
I20260923 17:25:22.442301 12385 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:22.442461 12385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:22.442773 12385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:22.445851 12385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "71d6ef19f6374f92932d361eb5c86dff"
format_stamp: "Formatted at 2026-09-23 17:25:22 on dist-test-slave-j0m4"
server_key: "c26f536d64de38265a2c14cc3fd576b6"
server_key_iv: "1afd5340d6bc5d8eb40845c5c887c9bd"
server_key_version: "encryptionkey@0"
I20260923 17:25:22.446669 12385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "71d6ef19f6374f92932d361eb5c86dff"
format_stamp: "Formatted at 2026-09-23 17:25:22 on dist-test-slave-j0m4"
server_key: "c26f536d64de38265a2c14cc3fd576b6"
server_key_iv: "1afd5340d6bc5d8eb40845c5c887c9bd"
server_key_version: "encryptionkey@0"
I20260923 17:25:22.452111 12385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:25:22.455613 12400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:22.456902 12385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 17:25:22.457095 12385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "71d6ef19f6374f92932d361eb5c86dff"
format_stamp: "Formatted at 2026-09-23 17:25:22 on dist-test-slave-j0m4"
server_key: "c26f536d64de38265a2c14cc3fd576b6"
server_key_iv: "1afd5340d6bc5d8eb40845c5c887c9bd"
server_key_version: "encryptionkey@0"
I20260923 17:25:22.457302 12385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:22.473712 12385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:22.478173 12385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:22.478487 12385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:22.479571 12385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:22.480962 12385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:22.481034 12385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:22.481206 12385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:22.481258 12385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:22.482295 12245 heartbeater.cc:503] Master 127.28.72.254:46851 was elected leader, sending a full tablet report...
I20260923 17:25:22.515622 12385 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:37251
I20260923 17:25:22.515650 12512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:37251 every 8 connection(s)
I20260923 17:25:22.517017 12385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:25:22.518894 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 12385
I20260923 17:25:22.519137 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:25:22.519765 28963 external_mini_cluster.cc:1428] Setting key e84579474ef4120c70063ee615ff5c9c
Sep 23 17:25:22 dist-test-slave-j0m4 krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184322, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:25:22.539028 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.519727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:40017 (local address 127.28.72.254:46851)
0923 17:25:22.519919 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:22.519932 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:22.521475 (+ 1543us) server_negotiation.cc:407] Connection header received
0923 17:25:22.523811 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:22.523816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:22.523910 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:22.524026 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:22.525606 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.526402 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.528113 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.528528 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.528660 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.532352 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:22.532358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:22.532361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:22.532397 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:22.535793 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:22.536757 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:22.536763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:22.536767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:22.536931 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:22.537402 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:22.537405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:22.537407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:22.537724 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:22.537841 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:22.538603 (+ 762us) server_negotiation.cc:299] Negotiation successful
0923 17:25:22.538783 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:25:22.540127 12515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.520480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:40017)
0923 17:25:22.521324 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:22.521371 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:22.523530 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:22.524282 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:22.524291 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:22.524835 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:22.525436 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.525454 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.526583 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.526587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.527960 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.527972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.528768 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.528775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.529014 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.529746 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:22.529770 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:22.532092 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:22.535970 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:22.535982 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:22.536005 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:22.536621 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:22.537072 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:22.537076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:22.537079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:22.537291 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:22.537830 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:22.537836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:22.538258 (+ 422us) client_negotiation.cc:769] Sending connection context
0923 17:25:22.538500 (+ 242us) client_negotiation.cc:240] Negotiation successful
0923 17:25:22.538909 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":323,"threads_started":1}
I20260923 17:25:22.541556 12513 heartbeater.cc:347] Connected to a master server at 127.28.72.254:46851
I20260923 17:25:22.541867 12513 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:22.542619 12513 heartbeater.cc:511] Master 127.28.72.254:46851 requested a full tablet report, sending...
I20260923 17:25:22.544142 12056 ts_manager.cc:194] Registered new tserver with Master: 71d6ef19f6374f92932d361eb5c86dff (127.28.72.195:37251)
I20260923 17:25:22.544970 12056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:40017
I20260923 17:25:22.548518 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:25:22.567349 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.553315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.28.72.254:46851)
0923 17:25:22.553565 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:22.553574 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:22.553891 (+ 317us) server_negotiation.cc:407] Connection header received
0923 17:25:22.554088 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:22.554095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:22.554180 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:22.554311 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:22.555887 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.556633 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.558012 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.558473 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.558619 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.561332 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:22.561338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:22.561341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:22.561375 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:22.564323 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:22.565144 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:22.565156 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:22.565158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:22.565337 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:22.565747 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:22.565753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:22.565757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:22.566105 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:22.566382 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:22.566733 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 17:25:22.566975 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 17:25:22.571969 12056 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:25:22.579263 12056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:25:22.604415 12525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.592885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:39849 (local address 127.0.0.1:40608)
0923 17:25:22.594775 (+ 1890us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:22.594825 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:22.596334 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:22.597169 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:22.597180 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:22.597222 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:22.597679 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.597693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.599710 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.599714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.601747 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.601759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.603284 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.603290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.603499 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.603554 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 17:25:22.603689 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 17:25:22.603873 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1641,"mutex_wait_us":1,"thread_start_us":1497,"threads_started":1}
I20260923 17:25:22.605733 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.594026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.28.72.194:44777)
0923 17:25:22.595443 (+ 1417us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:22.595453 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:22.596965 (+ 1512us) server_negotiation.cc:407] Connection header received
0923 17:25:22.597076 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:22.597083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:22.597291 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:22.597476 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:22.598421 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.600021 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.602761 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.604321 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.604456 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.604644 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:22.605221 (+ 577us) server_negotiation.cc:299] Negotiation successful
0923 17:25:22.605403 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1207,"thread_start_us":1103,"threads_started":1}
I20260923 17:25:22.605746 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.593209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.28.72.193:39849)
0923 17:25:22.594360 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:22.594372 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:22.596180 (+ 1808us) server_negotiation.cc:407] Connection header received
0923 17:25:22.596625 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:22.596634 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:22.596932 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:22.597119 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:22.597887 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.599383 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.601903 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.603024 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.603242 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.603500 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:22.603750 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 17:25:22.605425 (+ 1675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":818,"threads_started":1}
I20260923 17:25:22.605775 12527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.593345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:44777 (local address 127.0.0.1:40820)
0923 17:25:22.596716 (+ 3371us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:22.596731 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:22.596833 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:22.597748 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:22.597753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:22.597778 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:22.598244 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.598260 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.600176 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.600181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.602590 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.602604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.604784 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.604791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.605035 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.605059 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:25:22.605167 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 17:25:22.605298 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2418,"mutex_wait_us":3,"thread_start_us":2247,"threads_started":1}
I20260923 17:25:22.605841 12529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.595149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:37251 (local address 127.0.0.1:55290)
0923 17:25:22.596302 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:22.596322 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:22.596495 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:22.598370 (+ 1875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:22.598376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:22.598401 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:22.598742 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.598752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.600644 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.600649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.602843 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.602852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.604925 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.604934 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.605124 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.605174 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 17:25:22.605260 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 17:25:22.605392 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":1,"thread_start_us":823,"threads_started":1}
I20260923 17:25:22.606631 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.595485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.28.72.195:37251)
0923 17:25:22.597816 (+ 2331us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:22.597825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:22.597858 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:25:22.597948 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:22.597957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:22.598169 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:22.598357 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:22.598935 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.600471 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.603000 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.604466 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.604700 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.604915 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:22.606143 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0923 17:25:22.606311 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2154,"thread_start_us":1987,"threads_started":1}
I20260923 17:25:22.610181 12448 tablet_service.cc:1467] Processing CreateTablet for tablet c811c0ecd0494e2780d14cb8d85989cc (DEFAULT_TABLE table=test-table [id=faa81e715ccb42529d5a1f5ac708ec32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:22.610179 12180 tablet_service.cc:1467] Processing CreateTablet for tablet c811c0ecd0494e2780d14cb8d85989cc (DEFAULT_TABLE table=test-table [id=faa81e715ccb42529d5a1f5ac708ec32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:22.610179 12314 tablet_service.cc:1467] Processing CreateTablet for tablet c811c0ecd0494e2780d14cb8d85989cc (DEFAULT_TABLE table=test-table [id=faa81e715ccb42529d5a1f5ac708ec32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:22.611618 12448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c811c0ecd0494e2780d14cb8d85989cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:22.611631 12180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c811c0ecd0494e2780d14cb8d85989cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:22.611653 12314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c811c0ecd0494e2780d14cb8d85989cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:22.624768 12531 tablet_bootstrap.cc:492] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff: Bootstrap starting.
I20260923 17:25:22.627835 12533 tablet_bootstrap.cc:492] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de: Bootstrap starting.
I20260923 17:25:22.628782 12531 tablet_bootstrap.cc:654] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:22.629567 12532 tablet_bootstrap.cc:492] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460: Bootstrap starting.
I20260923 17:25:22.630627 12531 log.cc:824] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:22.631526 12533 tablet_bootstrap.cc:654] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:22.632833 12532 tablet_bootstrap.cc:654] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:22.632843 12533 log.cc:824] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:22.634027 12532 log.cc:824] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:22.634416 12531 tablet_bootstrap.cc:492] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff: No bootstrap required, opened a new log
I20260923 17:25:22.634703 12531 ts_tablet_manager.cc:1397] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260923 17:25:22.636092 12533 tablet_bootstrap.cc:492] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de: No bootstrap required, opened a new log
I20260923 17:25:22.636379 12533 ts_tablet_manager.cc:1397] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 17:25:22.636752 12532 tablet_bootstrap.cc:492] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460: No bootstrap required, opened a new log
I20260923 17:25:22.637005 12532 ts_tablet_manager.cc:1397] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260923 17:25:22.644841 12532 raft_consensus.cc:348] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } } peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } }
I20260923 17:25:22.645278 12532 raft_consensus.cc:374] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:22.645375 12532 raft_consensus.cc:729] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d0fd247bf44f9fba3197863d854460, State: Initialized, Role: FOLLOWER
I20260923 17:25:22.645788 12531 raft_consensus.cc:348] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } } peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } }
I20260923 17:25:22.646045 12532 consensus_queue.cc:260] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } } peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } }
I20260923 17:25:22.646248 12531 raft_consensus.cc:374] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:22.646347 12531 raft_consensus.cc:729] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d6ef19f6374f92932d361eb5c86dff, State: Initialized, Role: FOLLOWER
I20260923 17:25:22.647054 12531 consensus_queue.cc:260] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } } peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } }
I20260923 17:25:22.647406 12379 heartbeater.cc:503] Master 127.28.72.254:46851 was elected leader, sending a full tablet report...
I20260923 17:25:22.647976 12532 ts_tablet_manager.cc:1428] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260923 17:25:22.649072 12533 raft_consensus.cc:348] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } } peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } }
I20260923 17:25:22.649652 12533 raft_consensus.cc:374] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:22.649735 12533 raft_consensus.cc:729] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f394a6ef0349e9b56fae2c8302c9de, State: Initialized, Role: FOLLOWER
I20260923 17:25:22.650171 12513 heartbeater.cc:503] Master 127.28.72.254:46851 was elected leader, sending a full tablet report...
I20260923 17:25:22.650513 12533 consensus_queue.cc:260] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } } peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } }
I20260923 17:25:22.651010 12531 ts_tablet_manager.cc:1428] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260923 17:25:22.653375 12533 ts_tablet_manager.cc:1428] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
W20260923 17:25:22.698496 12246 tablet.cc:2369] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:25:22.752753 12380 tablet.cc:2369] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:25:22.768676 12514 tablet.cc:2369] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:25:22.796762 12537 raft_consensus.cc:482] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:25:22.797096 12537 raft_consensus.cc:504] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } } peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } }
I20260923 17:25:22.799036 12537 leader_election.cc:288] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09f394a6ef0349e9b56fae2c8302c9de (127.28.72.193:39849), 71d6ef19f6374f92932d361eb5c86dff (127.28.72.195:37251)
I20260923 17:25:22.806237 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.799741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:33079 (local address 127.28.72.195:37251)
0923 17:25:22.799987 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:22.799996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:22.800681 (+ 685us) server_negotiation.cc:407] Connection header received
0923 17:25:22.800880 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:22.800892 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:22.800960 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:22.801065 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:22.801685 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.802441 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.804390 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.805219 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.805424 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.805506 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:22.805820 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 17:25:22.805952 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:25:22.806275 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.799475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:44783 (local address 127.28.72.193:39849)
0923 17:25:22.799723 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:22.799731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:22.800461 (+ 730us) server_negotiation.cc:407] Connection header received
0923 17:25:22.800698 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:22.800705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:22.800772 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:22.800867 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:22.801465 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.802216 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.804339 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.805309 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.805525 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.805603 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:22.805875 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 17:25:22.806010 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 17:25:22.806440 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.799399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:39849 (local address 127.28.72.194:44783)
0923 17:25:22.800326 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:22.800345 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:22.800542 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:22.800898 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:22.800902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:22.800947 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:22.801306 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.801319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.802353 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.802360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.804152 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.804170 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.805561 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.805568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.805718 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.805741 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:25:22.805847 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 17:25:22.806000 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":5,"thread_start_us":513,"threads_started":1}
I20260923 17:25:22.806895 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.799460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:37251 (local address 127.28.72.194:33079)
0923 17:25:22.800559 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:22.800577 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:22.800729 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:22.801084 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:22.801089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:22.801113 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:22.801529 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.801538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.802589 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.802596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.804209 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.804221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.805469 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.805476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.805655 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.805679 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:25:22.805788 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 17:25:22.805933 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":31,"thread_start_us":715,"threads_started":1}
I20260923 17:25:22.807837 12200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c811c0ecd0494e2780d14cb8d85989cc" candidate_uuid: "89d0fd247bf44f9fba3197863d854460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f394a6ef0349e9b56fae2c8302c9de" is_pre_election: true
I20260923 17:25:22.807976 12468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c811c0ecd0494e2780d14cb8d85989cc" candidate_uuid: "89d0fd247bf44f9fba3197863d854460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d6ef19f6374f92932d361eb5c86dff" is_pre_election: true
I20260923 17:25:22.808493 12468 raft_consensus.cc:2454] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89d0fd247bf44f9fba3197863d854460 in term 0.
I20260923 17:25:22.808493 12200 raft_consensus.cc:2454] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89d0fd247bf44f9fba3197863d854460 in term 0.
I20260923 17:25:22.809249 12267 leader_election.cc:302] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d6ef19f6374f92932d361eb5c86dff, 89d0fd247bf44f9fba3197863d854460; no voters:
I20260923 17:25:22.809630 12537 raft_consensus.cc:2781] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:25:22.809798 12537 raft_consensus.cc:482] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:25:22.809917 12537 raft_consensus.cc:3037] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:22.812211 12537 raft_consensus.cc:504] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } } peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } }
I20260923 17:25:22.812872 12537 leader_election.cc:288] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [CANDIDATE]: Term 1 election: Requested vote from peers 09f394a6ef0349e9b56fae2c8302c9de (127.28.72.193:39849), 71d6ef19f6374f92932d361eb5c86dff (127.28.72.195:37251)
I20260923 17:25:22.813423 12200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c811c0ecd0494e2780d14cb8d85989cc" candidate_uuid: "89d0fd247bf44f9fba3197863d854460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f394a6ef0349e9b56fae2c8302c9de"
I20260923 17:25:22.813422 12468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c811c0ecd0494e2780d14cb8d85989cc" candidate_uuid: "89d0fd247bf44f9fba3197863d854460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d6ef19f6374f92932d361eb5c86dff"
I20260923 17:25:22.813704 12200 raft_consensus.cc:3037] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:22.813704 12468 raft_consensus.cc:3037] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:22.817013 12468 raft_consensus.cc:2454] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89d0fd247bf44f9fba3197863d854460 in term 1.
I20260923 17:25:22.817013 12200 raft_consensus.cc:2454] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89d0fd247bf44f9fba3197863d854460 in term 1.
I20260923 17:25:22.817494 12270 leader_election.cc:302] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f394a6ef0349e9b56fae2c8302c9de, 89d0fd247bf44f9fba3197863d854460; no voters:
I20260923 17:25:22.817852 12537 raft_consensus.cc:2781] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:22.818328 12537 raft_consensus.cc:686] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [term 1 LEADER]: Becoming Leader. State: Replica: 89d0fd247bf44f9fba3197863d854460, State: Running, Role: LEADER
I20260923 17:25:22.818823 12537 consensus_queue.cc:237] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } } peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } }
I20260923 17:25:22.823879 12056 catalog_manager.cc:5495] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89d0fd247bf44f9fba3197863d854460 (127.28.72.194). New cstate: current_term: 1 leader_uuid: "89d0fd247bf44f9fba3197863d854460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } health_report { overall_health: UNKNOWN } } }
I20260923 17:25:22.842226 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.836761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.28.72.194:44777)
0923 17:25:22.837032 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:22.837041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:22.837075 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:25:22.837174 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:22.837178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:22.837261 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:22.837418 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:22.837796 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.838567 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.840033 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.840438 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.840576 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.841094 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:22.841605 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:22.841764 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:22.841851 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 17:25:22.841935 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 17:25:22.850807 12200 raft_consensus.cc:1261] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de [term 1 FOLLOWER]: Refusing update from remote peer 89d0fd247bf44f9fba3197863d854460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:22.850807 12468 raft_consensus.cc:1261] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff [term 1 FOLLOWER]: Refusing update from remote peer 89d0fd247bf44f9fba3197863d854460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:22.851939 12537 consensus_queue.cc:1035] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:22.852244 12542 consensus_queue.cc:1035] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:22.880987 12055 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:22.881294 12055 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:22.885269 12055 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Sending DeleteTablet for 3 replicas of tablet c811c0ecd0494e2780d14cb8d85989cc
I20260923 17:25:22.886574 12314 tablet_service.cc:1514] Processing DeleteTablet for tablet c811c0ecd0494e2780d14cb8d85989cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:22 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:40820
I20260923 17:25:22.886574 12180 tablet_service.cc:1514] Processing DeleteTablet for tablet c811c0ecd0494e2780d14cb8d85989cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:22 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:40608
I20260923 17:25:22.886859 12448 tablet_service.cc:1514] Processing DeleteTablet for tablet c811c0ecd0494e2780d14cb8d85989cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:22 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:55290
I20260923 17:25:22.887161 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 12117
I20260923 17:25:22.899283 12556 tablet_replica.cc:330] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460: stopping tablet replica
I20260923 17:25:22.900012 12555 tablet_replica.cc:330] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de: stopping tablet replica
I20260923 17:25:22.900760 12555 raft_consensus.cc:2229] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:22.901374 12556 raft_consensus.cc:2229] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:25:22.902066 12555 raft_consensus.cc:2258] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:22.902814 12556 raft_consensus.cc:2258] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:22.905684 12556 ts_tablet_manager.cc:1905] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:22.905684 12555 ts_tablet_manager.cc:1905] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:22.907444 12557 tablet_replica.cc:330] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff: stopping tablet replica
I20260923 17:25:22.907955 12557 raft_consensus.cc:2229] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:22.908492 12557 raft_consensus.cc:2258] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:22.910863 12557 ts_tablet_manager.cc:1905] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:22.912935 12555 ts_tablet_manager.cc:1918] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:22.913157 12555 log.cc:1196] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/c811c0ecd0494e2780d14cb8d85989cc
I20260923 17:25:22.913450 12556 ts_tablet_manager.cc:1918] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:22.913726 12556 log.cc:1196] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/c811c0ecd0494e2780d14cb8d85989cc
I20260923 17:25:22.913761 12555 ts_tablet_manager.cc:1939] T c811c0ecd0494e2780d14cb8d85989cc P 09f394a6ef0349e9b56fae2c8302c9de: Deleting consensus metadata
Sep 23 17:25:22 dist-test-slave-j0m4 krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:25:22.914342 12556 ts_tablet_manager.cc:1939] T c811c0ecd0494e2780d14cb8d85989cc P 89d0fd247bf44f9fba3197863d854460: Deleting consensus metadata
I20260923 17:25:22.916220 12042 catalog_manager.cc:4841] TS 89d0fd247bf44f9fba3197863d854460 (127.28.72.194:44777): tablet c811c0ecd0494e2780d14cb8d85989cc (table test-table [id=faa81e715ccb42529d5a1f5ac708ec32]) successfully deleted
I20260923 17:25:22.918532 12557 ts_tablet_manager.cc:1918] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:22.918717 12557 log.cc:1196] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/c811c0ecd0494e2780d14cb8d85989cc
I20260923 17:25:22.919312 12557 ts_tablet_manager.cc:1939] T c811c0ecd0494e2780d14cb8d85989cc P 71d6ef19f6374f92932d361eb5c86dff: Deleting consensus metadata
I20260923 17:25:22.919471 12043 catalog_manager.cc:4841] TS 09f394a6ef0349e9b56fae2c8302c9de (127.28.72.193:39849): tablet c811c0ecd0494e2780d14cb8d85989cc (table test-table [id=faa81e715ccb42529d5a1f5ac708ec32]) successfully deleted
I20260923 17:25:22.921223 12040 catalog_manager.cc:4841] TS 71d6ef19f6374f92932d361eb5c86dff (127.28.72.195:37251): tablet c811c0ecd0494e2780d14cb8d85989cc (table test-table [id=faa81e715ccb42529d5a1f5ac708ec32]) successfully deleted
I20260923 17:25:22.922982 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.888951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.28.72.193:39849)
0923 17:25:22.889216 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:22.889225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:22.899929 (+ 10704us) server_negotiation.cc:407] Connection header received
0923 17:25:22.900596 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:22.900606 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:22.900731 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:22.900837 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:22.903900 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.905495 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.908603 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.909263 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.909386 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.915189 (+ 5803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:22.915206 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:22.915225 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:22.915274 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:22.919571 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:22.920569 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:22.920579 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:22.920583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:22.920748 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:22.921250 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:22.921255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:22.921259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:22.921753 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:22.922075 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:22.922347 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 17:25:22.922593 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:25:22.923770 12240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:23.015375 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 12117
I20260923 17:25:23.034552 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 12251
Sep 23 17:25:23 dist-test-slave-j0m4 krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:25:23.053496 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:23.035523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.28.72.194:44777)
0923 17:25:23.035735 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:23.035744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:23.035797 (+ 53us) server_negotiation.cc:407] Connection header received
0923 17:25:23.035863 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:23.035868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:23.035950 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:23.036054 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:23.037593 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.038872 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.040298 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.041018 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.041138 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:23.046050 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:23.046069 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:23.046089 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:23.046144 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:23.050232 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:23.051062 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:23.051072 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:23.051076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:23.051277 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:23.051772 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:23.051777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:23.051781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:23.052240 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:23.052591 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:23.052828 (+ 237us) server_negotiation.cc:299] Negotiation successful
0923 17:25:23.053085 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:25:23.054343 12374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:23.169770 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 12251
I20260923 17:25:23.189024 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 12385
Sep 23 17:25:23 dist-test-slave-j0m4 krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:25:23.207327 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:23.189759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.28.72.195:37251)
0923 17:25:23.189970 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:23.190011 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:23.190036 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:25:23.190237 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:23.190242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:23.190312 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:23.190392 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:23.191966 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.192727 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.194131 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.194583 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.194692 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:23.200328 (+ 5636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:23.200347 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:23.200364 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:23.200417 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:23.204277 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:23.205291 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:23.205299 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:23.205301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:23.205475 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:23.205826 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:23.205829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:23.205831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:23.206225 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:23.206447 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:23.206707 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 17:25:23.206880 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:25:23.208191 12508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:23.298032 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 12385
I20260923 17:25:23.315675 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 12024
I20260923 17:25:23.316937 12086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:23.429260 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 12024
I20260923 17:25:43.453429 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:46851
--webserver_interface=127.28.72.254
--webserver_port=34121
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:46851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:43.803454 12563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:43.803789 12563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:43.803987 12563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:43.811872 12563 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:25:43.811985 12563 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:25:43.812059 12563 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:25:43.812144 12563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:43.812247 12563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:43.812389 12563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:43.812453 12563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:43.812525 12563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:25:43.823076 12563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:46851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:46851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=34121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:25:43.824559 12563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:43.825975 12563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:43.851200 12572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:43.851466 12569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:43.851221 12570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:43.852066 12563 server_base.cc:1034] running on GCE node
I20260923 17:25:43.853205 12563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:43.855031 12563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:43.856369 12563 hybrid_clock.cc:648] HybridClock initialized: now 1790184343856291 us; error 51 us; skew 500 ppm
I20260923 17:25:43.856846 12563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:25:43 dist-test-slave-j0m4 krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184343, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:43.861089 12563 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:25:43.863081 12563 webserver.cc:460] Webserver started at http://127.28.72.254:34121/ using document root <none> and password file <none>
I20260923 17:25:43.864125 12563 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:43.864238 12563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:43.870983 12563 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:25:43.875849 12579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:43.877614 12563 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:25:43.877890 12563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "4b9f217d69754c9f94cafce2891ef076"
format_stamp: "Formatted at 2026-09-23 17:25:20 on dist-test-slave-j0m4"
server_key: "2f2b066c943275468cd21afad03932e2"
server_key_iv: "b76db5170747720f893057c6a727d816"
server_key_version: "encryptionkey@0"
I20260923 17:25:43.879063 12563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:43.898751 12563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:43.903903 12563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:43.904240 12563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:43.924579 12563 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:46851
I20260923 17:25:43.924675 12630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:46851 every 8 connection(s)
I20260923 17:25:43.926262 12563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:25:43.929617 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 12563
I20260923 17:25:43.930678 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:39849
--local_ip_for_outbound_sockets=127.28.72.193
--tserver_master_addrs=127.28.72.254:46851
--webserver_port=44037
--webserver_interface=127.28.72.193
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:25:43.936522 12631 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:25:43.941699 12631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Bootstrap starting.
I20260923 17:25:43.962984 12631 log.cc:824] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:43.978084 12631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:25:43.978816 12631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Bootstrap complete.
I20260923 17:25:43.990242 12631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } }
I20260923 17:25:43.991341 12631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9f217d69754c9f94cafce2891ef076, State: Initialized, Role: FOLLOWER
I20260923 17:25:43.992162 12631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } }
I20260923 17:25:43.992449 12631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:25:43.992590 12631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:25:43.992820 12631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:25:43.997458 12631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } }
I20260923 17:25:43.998123 12631 leader_election.cc:302] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b9f217d69754c9f94cafce2891ef076; no voters:
I20260923 17:25:43.998894 12631 leader_election.cc:288] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:25:43.999049 12635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:25:44.000412 12635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 2 LEADER]: Becoming Leader. State: Replica: 4b9f217d69754c9f94cafce2891ef076, State: Running, Role: LEADER
I20260923 17:25:44.001178 12635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } }
I20260923 17:25:44.001807 12631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:44.004467 12637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b9f217d69754c9f94cafce2891ef076. Latest consensus state: current_term: 2 leader_uuid: "4b9f217d69754c9f94cafce2891ef076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } } }
I20260923 17:25:44.004881 12637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:44.004506 12636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4b9f217d69754c9f94cafce2891ef076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } } }
I20260923 17:25:44.005040 12636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:44.016860 12641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:25:44.028061 12641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:25:44.030395 12641 catalog_manager.cc:1259] Loaded cluster ID: efb6904c87e249dc996d0839a97fec2c
I20260923 17:25:44.030681 12641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:25:44.034940 12652 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:25:44.036284 12641 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:25:44.037727 12641 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Loaded TSK: 0
I20260923 17:25:44.068146 12641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Generated new TSK 1
I20260923 17:25:44.068650 12641 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:25:44.338531 12633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:44.338912 12633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:44.339080 12633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:44.347329 12633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:44.347515 12633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:44.347697 12633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:25:44.358665 12633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:39849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=44037
--tserver_master_addrs=127.28.72.254:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:25:44.360311 12633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:44.361801 12633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:44.377205 12658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:44.377347 12659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:44.377768 12661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:44.378150 12633 server_base.cc:1034] running on GCE node
I20260923 17:25:44.378904 12633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:44.380196 12633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:44.381441 12633 hybrid_clock.cc:648] HybridClock initialized: now 1790184344381390 us; error 45 us; skew 500 ppm
I20260923 17:25:44.381803 12633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:25:44 dist-test-slave-j0m4 krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184344, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:44.385514 12633 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:25:44.387367 12633 webserver.cc:460] Webserver started at http://127.28.72.193:44037/ using document root <none> and password file <none>
I20260923 17:25:44.388126 12633 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:44.388279 12633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:44.394339 12633 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260923 17:25:44.397485 12668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:44.399262 12633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:25:44.399458 12633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "09f394a6ef0349e9b56fae2c8302c9de"
format_stamp: "Formatted at 2026-09-23 17:25:21 on dist-test-slave-j0m4"
server_key: "9ebaac2a4080c4e56b103b4876fe1f78"
server_key_iv: "cae692ef66ac42a81302654aea67a915"
server_key_version: "encryptionkey@0"
I20260923 17:25:44.400496 12633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:44.426561 12633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:44.431769 12633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:44.432073 12633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:44.433131 12633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:44.434561 12633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:44.434659 12633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:44.434811 12633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:44.434908 12633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:44.470839 12633 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:39849
I20260923 17:25:44.471012 12780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:39849 every 8 connection(s)
I20260923 17:25:44.472522 12633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:25:44.482872 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 12633
I20260923 17:25:44.483773 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:44777
--local_ip_for_outbound_sockets=127.28.72.194
--tserver_master_addrs=127.28.72.254:46851
--webserver_port=33629
--webserver_interface=127.28.72.194
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:25:44 dist-test-slave-j0m4 krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184344, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:25:44.500018 12783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:44.476019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:35649 (local address 127.28.72.254:46851)
0923 17:25:44.476741 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:44.476755 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:44.477527 (+ 772us) server_negotiation.cc:407] Connection header received
0923 17:25:44.479884 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:44.479899 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:44.480335 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:44.480690 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:44.482275 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:44.484668 (+ 2393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:44.486365 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:44.487652 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:44.487993 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:44.491850 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:44.491860 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:44.491878 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:44.491920 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:44.495046 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:44.495945 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:44.495956 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:44.495971 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:44.496143 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:44.496625 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:44.496635 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:44.496637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:44.497150 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:44.497360 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:44.498202 (+ 842us) server_negotiation.cc:299] Negotiation successful
0923 17:25:44.498617 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":347,"threads_started":1}
I20260923 17:25:44.500018 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:44.476368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:35649)
0923 17:25:44.477310 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:44.477365 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:44.479260 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:44.480834 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:44.480842 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:44.481397 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:44.481970 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:44.481988 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:44.484888 (+ 2900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:44.484893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:44.486193 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:44.486208 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:44.487887 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:44.487892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:44.488095 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:44.488867 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:44.488894 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:44.491512 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:44.495291 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:44.495302 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:44.495320 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:44.495783 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:44.496276 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:44.496281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:44.496285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:44.496490 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:44.497308 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:44.497317 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:44.497681 (+ 364us) client_negotiation.cc:769] Sending connection context
0923 17:25:44.497981 (+ 300us) client_negotiation.cc:240] Negotiation successful
0923 17:25:44.498586 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":383,"threads_started":1}
I20260923 17:25:44.503350 12781 heartbeater.cc:347] Connected to a master server at 127.28.72.254:46851
I20260923 17:25:44.503722 12781 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:44.504704 12781 heartbeater.cc:511] Master 127.28.72.254:46851 requested a full tablet report, sending...
I20260923 17:25:44.507085 12596 ts_manager.cc:194] Registered new tserver with Master: 09f394a6ef0349e9b56fae2c8302c9de (127.28.72.193:39849)
I20260923 17:25:44.508654 12596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:35649
W20260923 17:25:44.829478 12785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:44.829773 12785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:44.829937 12785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:44.837388 12785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:44.837519 12785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:44.837715 12785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:25:44.847869 12785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:44777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=33629
--tserver_master_addrs=127.28.72.254:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:25:44.849292 12785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:44.850749 12785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:44.865200 12790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:44.865175 12793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:44.865681 12791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:44.866318 12785 server_base.cc:1034] running on GCE node
I20260923 17:25:44.866981 12785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:44.868064 12785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:44.869338 12785 hybrid_clock.cc:648] HybridClock initialized: now 1790184344869294 us; error 38 us; skew 500 ppm
I20260923 17:25:44.869711 12785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:25:44 dist-test-slave-j0m4 krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184344, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:44.873337 12785 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:25:44.875003 12785 webserver.cc:460] Webserver started at http://127.28.72.194:33629/ using document root <none> and password file <none>
I20260923 17:25:44.876026 12785 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:44.876194 12785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:44.882866 12785 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 17:25:44.886341 12800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:44.887964 12785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:25:44.888159 12785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "89d0fd247bf44f9fba3197863d854460"
format_stamp: "Formatted at 2026-09-23 17:25:21 on dist-test-slave-j0m4"
server_key: "48bac22c6e255f610380e9e158943c69"
server_key_iv: "6f9f31754f6c9089c6ea99987d3c2097"
server_key_version: "encryptionkey@0"
I20260923 17:25:44.889207 12785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:44.906287 12785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:44.910939 12785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:44.911312 12785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:44.912335 12785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:44.913828 12785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:44.913951 12785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:44.914086 12785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:44.914186 12785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:44.966220 12785 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:44777
I20260923 17:25:44.966387 12912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:44777 every 8 connection(s)
I20260923 17:25:44.967881 12785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:25:44.971342 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 12785
I20260923 17:25:44.972209 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:37251
--local_ip_for_outbound_sockets=127.28.72.195
--tserver_master_addrs=127.28.72.254:46851
--webserver_port=36601
--webserver_interface=127.28.72.195
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:25:44 dist-test-slave-j0m4 krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184344, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:25:44.990275 12783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:44.970899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:44505 (local address 127.28.72.254:46851)
0923 17:25:44.971161 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:44.971168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:44.972856 (+ 1688us) server_negotiation.cc:407] Connection header received
0923 17:25:44.974943 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:44.974948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:44.975072 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:44.975225 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:44.976922 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:44.977659 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:44.979410 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:44.979867 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:44.980012 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:44.983772 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:44.983777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:44.983780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:44.983816 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:44.986646 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:44.987599 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:44.987604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:44.987606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:44.987737 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:44.988282 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:44.988292 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:44.988294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:44.988589 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:44.988755 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:44.989649 (+ 894us) server_negotiation.cc:299] Negotiation successful
0923 17:25:44.989919 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:25:44.991405 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:44.971592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:44505)
0923 17:25:44.972661 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:44.972710 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:44.974625 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:44.975480 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:44.975490 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:44.976086 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:44.976716 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:44.976736 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:44.977832 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:44.977842 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:44.979243 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:44.979256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:44.980120 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:44.980127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:44.980377 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:44.981113 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:44.981137 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:44.983504 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:44.986839 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:44.986850 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:44.986872 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:44.987448 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:44.987862 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:44.987867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:44.987871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:44.988155 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:44.988725 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:44.988734 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:44.989248 (+ 514us) client_negotiation.cc:769] Sending connection context
0923 17:25:44.989536 (+ 288us) client_negotiation.cc:240] Negotiation successful
0923 17:25:44.990123 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":410,"threads_started":1}
I20260923 17:25:44.993196 12913 heartbeater.cc:347] Connected to a master server at 127.28.72.254:46851
I20260923 17:25:44.993526 12913 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:44.994509 12913 heartbeater.cc:511] Master 127.28.72.254:46851 requested a full tablet report, sending...
I20260923 17:25:44.996156 12596 ts_manager.cc:194] Registered new tserver with Master: 89d0fd247bf44f9fba3197863d854460 (127.28.72.194:44777)
I20260923 17:25:44.997108 12596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:44505
W20260923 17:25:45.319525 12916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:45.319876 12916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:45.320046 12916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:45.327466 12916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:45.327637 12916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:45.327818 12916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:25:45.337807 12916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:37251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=36601
--tserver_master_addrs=127.28.72.254:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:25:45.339254 12916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:45.340636 12916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:45.354722 12921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:45.354761 12924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:45.354802 12922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:45.355758 12916 server_base.cc:1034] running on GCE node
I20260923 17:25:45.356499 12916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:45.357533 12916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:45.358817 12916 hybrid_clock.cc:648] HybridClock initialized: now 1790184345358700 us; error 75 us; skew 500 ppm
I20260923 17:25:45.359277 12916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:25:45 dist-test-slave-j0m4 krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184345, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:45.362874 12916 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:25:45.364811 12916 webserver.cc:460] Webserver started at http://127.28.72.195:36601/ using document root <none> and password file <none>
I20260923 17:25:45.365618 12916 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:45.365744 12916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:45.371853 12916 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 17:25:45.374867 12931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:45.376505 12916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:25:45.376722 12916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "71d6ef19f6374f92932d361eb5c86dff"
format_stamp: "Formatted at 2026-09-23 17:25:22 on dist-test-slave-j0m4"
server_key: "c26f536d64de38265a2c14cc3fd576b6"
server_key_iv: "1afd5340d6bc5d8eb40845c5c887c9bd"
server_key_version: "encryptionkey@0"
I20260923 17:25:45.377810 12916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:45.408668 12916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:45.413254 12916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:45.413560 12916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:45.414659 12916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:45.416105 12916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:45.416193 12916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:45.416386 12916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:45.416483 12916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:45.451257 12916 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:37251
I20260923 17:25:45.451309 13043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:37251 every 8 connection(s)
I20260923 17:25:45.452816 12916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:25:45.459443 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 12916
Sep 23 17:25:45 dist-test-slave-j0m4 krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184345, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:25:45.479947 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.460553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60134 (local address 127.28.72.254:46851)
0923 17:25:45.461050 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:45.461055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:45.461510 (+ 455us) server_negotiation.cc:407] Connection header received
0923 17:25:45.461682 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:45.461685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:45.461748 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:45.461840 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:45.463866 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.464962 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.466809 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.467405 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.467555 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.470636 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:45.470643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:45.470648 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:45.470725 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:45.476255 (+ 5530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:45.477125 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:45.477132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:45.477136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:45.477320 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:45.477981 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:45.477985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:45.477987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:45.478453 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:45.478733 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:45.479243 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 17:25:45.479442 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":151,"server-negotiator.queue_time_us":381,"spinlock_wait_cycles":60800,"thread_start_us":263,"threads_started":1}
I20260923 17:25:45.480445 12783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.455748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:57479 (local address 127.28.72.254:46851)
0923 17:25:45.455946 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:45.455950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:45.457161 (+ 1211us) server_negotiation.cc:407] Connection header received
0923 17:25:45.459363 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:45.459369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:45.459451 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:45.459592 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:45.461621 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.462908 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.465170 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.465835 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.465982 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.470649 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:45.470653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:45.470656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:45.470724 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:45.476216 (+ 5492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:45.477195 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:45.477199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:45.477202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:45.477319 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:45.478138 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:45.478143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:45.478146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:45.478453 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:45.478626 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:45.479636 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0923 17:25:45.480080 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":73,"spinlock_wait_cycles":8576}
I20260923 17:25:45.481494 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.456212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:57479)
0923 17:25:45.456981 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:45.457022 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:45.458828 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:45.459896 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:45.459908 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:45.460611 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:45.461404 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:45.461424 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.463084 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.463088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:45.464962 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:45.464991 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.466116 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.466122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:45.466387 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.467316 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:45.467343 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:45.470412 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:45.476440 (+ 6028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:45.476451 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:45.476473 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:45.477082 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:45.477632 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:45.477640 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:45.477644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:45.477974 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:45.478642 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:45.478651 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:45.479171 (+ 520us) client_negotiation.cc:769] Sending connection context
0923 17:25:45.479502 (+ 331us) client_negotiation.cc:240] Negotiation successful
0923 17:25:45.480020 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":296,"threads_started":1}
I20260923 17:25:45.483165 13044 heartbeater.cc:347] Connected to a master server at 127.28.72.254:46851
I20260923 17:25:45.483518 13044 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:45.484371 13044 heartbeater.cc:511] Master 127.28.72.254:46851 requested a full tablet report, sending...
I20260923 17:25:45.485870 12596 ts_manager.cc:194] Registered new tserver with Master: 71d6ef19f6374f92932d361eb5c86dff (127.28.72.195:37251)
I20260923 17:25:45.486716 12596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:57479
I20260923 17:25:45.495836 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:25:45.502943 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.496783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.28.72.254:46851)
0923 17:25:45.496994 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:45.496999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:45.497543 (+ 544us) server_negotiation.cc:407] Connection header received
0923 17:25:45.497703 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:45.497707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:45.497863 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:45.497993 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:45.498448 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.499242 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.500923 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.501353 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.501501 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.502070 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:45.502346 (+ 276us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 17:25:45.502719 (+ 373us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60144: token expired
Metrics: {"server-negotiator.queue_time_us":84}
W20260923 17:25:45.503087 13047 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60144: token expired
W20260923 17:25:45.503779 12518 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.72.254:46851)
I20260923 17:25:45.511909 12781 heartbeater.cc:503] Master 127.28.72.254:46851 was elected leader, sending a full tablet report...
I20260923 17:25:45.517783 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.504303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.28.72.254:46851)
0923 17:25:45.504451 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:45.504455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:45.504591 (+ 136us) server_negotiation.cc:407] Connection header received
0923 17:25:45.504765 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:45.504769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:45.504825 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:45.504898 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:45.506429 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.507190 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.508621 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.509060 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.509170 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.512160 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:45.512166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:45.512177 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:45.512213 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:45.514931 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:45.515644 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:45.515651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:45.515655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:45.515798 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:45.516246 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:45.516252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:45.516255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:45.516588 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:45.516874 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:45.517194 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 17:25:45.517420 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:25:45.521311 12596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:25:45.528678 12596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:25:45.572937 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.544037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.28.72.195:37251)
0923 17:25:45.563537 (+ 19500us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:45.563545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:45.563572 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:25:45.563649 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:45.563655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:45.563830 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:45.563990 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:45.565219 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.566533 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.569955 (+ 3422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.571548 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.571705 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.571955 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:45.572342 (+ 387us) server_negotiation.cc:299] Negotiation successful
0923 17:25:45.572536 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19323,"thread_start_us":19117,"threads_started":1}
I20260923 17:25:45.572955 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.543491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:37251 (local address 127.0.0.1:44408)
0923 17:25:45.556942 (+ 13451us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:45.557005 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:45.557259 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:45.564022 (+ 6763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:45.564026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:45.564052 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:45.564862 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:45.564876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.566705 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.566711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:45.569730 (+ 3019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:45.569754 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.571905 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.571911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:45.572135 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.572168 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:25:45.572258 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 17:25:45.572381 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13085,"mutex_wait_us":67,"thread_start_us":17232,"threads_started":1}
I20260923 17:25:45.572992 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.543875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:39849 (local address 127.0.0.1:37782)
0923 17:25:45.561036 (+ 17161us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:45.561062 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:45.561254 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:45.562801 (+ 1547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:45.562812 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:45.562852 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:45.565168 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:45.565184 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.567443 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.567448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:45.569996 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:45.570013 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.572123 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.572128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:45.572289 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.572322 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:25:45.572423 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 17:25:45.572562 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16999,"mutex_wait_us":414,"thread_start_us":17493,"threads_started":1}
I20260923 17:25:45.573462 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.546781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.28.72.194:44777)
0923 17:25:45.561982 (+ 15201us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:45.561991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:45.562021 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:25:45.562123 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:45.562130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:45.562485 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:45.563107 (+ 622us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:45.565390 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.567211 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.570134 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.571307 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.571485 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.571723 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:45.572846 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0923 17:25:45.573063 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15052,"thread_start_us":14918,"threads_started":1}
I20260923 17:25:45.573853 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.546051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:44777 (local address 127.0.0.1:57316)
0923 17:25:45.561348 (+ 15297us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:45.561365 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:45.561467 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:45.562801 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:45.562812 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:45.562875 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:45.565169 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:45.565188 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.567356 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.567361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:45.569949 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:45.569962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.571549 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.571556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:45.571780 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.571852 (+ 72us) client_negotiation.cc:769] Sending connection context
0923 17:25:45.572019 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 17:25:45.572268 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15170,"mutex_wait_us":126,"thread_start_us":10348,"threads_started":1}
I20260923 17:25:45.574796 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.544266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.28.72.193:39849)
0923 17:25:45.560623 (+ 16357us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:45.560637 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:45.561591 (+ 954us) server_negotiation.cc:407] Connection header received
0923 17:25:45.561733 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:45.561744 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:45.562046 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:45.562274 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:45.565365 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.567269 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.570204 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.571647 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.571775 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.571976 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:45.574221 (+ 2245us) server_negotiation.cc:299] Negotiation successful
0923 17:25:45.574447 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16097,"thread_start_us":15911,"threads_started":1}
I20260923 17:25:45.577490 12979 tablet_service.cc:1467] Processing CreateTablet for tablet cca0c64c70a24c92b70ef5d6e55c32df (DEFAULT_TABLE table=test-table [id=eda56ae165c74b119c89b8f5cfbafa73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:45.577893 12848 tablet_service.cc:1467] Processing CreateTablet for tablet cca0c64c70a24c92b70ef5d6e55c32df (DEFAULT_TABLE table=test-table [id=eda56ae165c74b119c89b8f5cfbafa73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:45.577943 12716 tablet_service.cc:1467] Processing CreateTablet for tablet cca0c64c70a24c92b70ef5d6e55c32df (DEFAULT_TABLE table=test-table [id=eda56ae165c74b119c89b8f5cfbafa73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:45.579367 12848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca0c64c70a24c92b70ef5d6e55c32df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:45.579433 12716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca0c64c70a24c92b70ef5d6e55c32df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:45.579985 12979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca0c64c70a24c92b70ef5d6e55c32df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:45.592041 13060 tablet_bootstrap.cc:492] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460: Bootstrap starting.
I20260923 17:25:45.595636 13060 tablet_bootstrap.cc:654] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:45.596875 13060 log.cc:824] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:45.597826 13062 tablet_bootstrap.cc:492] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff: Bootstrap starting.
I20260923 17:25:45.596961 13061 tablet_bootstrap.cc:492] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de: Bootstrap starting.
I20260923 17:25:45.600878 13060 tablet_bootstrap.cc:492] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460: No bootstrap required, opened a new log
I20260923 17:25:45.601174 13060 ts_tablet_manager.cc:1397] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 17:25:45.601715 13061 tablet_bootstrap.cc:654] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:45.602433 13062 tablet_bootstrap.cc:654] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:45.603065 13061 log.cc:824] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:45.603713 13062 log.cc:824] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:45.606215 13061 tablet_bootstrap.cc:492] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de: No bootstrap required, opened a new log
I20260923 17:25:45.606510 13061 ts_tablet_manager.cc:1397] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260923 17:25:45.606776 13062 tablet_bootstrap.cc:492] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff: No bootstrap required, opened a new log
I20260923 17:25:45.607043 13062 ts_tablet_manager.cc:1397] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260923 17:25:45.613135 13060 raft_consensus.cc:348] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } } peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } }
I20260923 17:25:45.613678 13060 raft_consensus.cc:374] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:45.613788 13060 raft_consensus.cc:729] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d0fd247bf44f9fba3197863d854460, State: Initialized, Role: FOLLOWER
I20260923 17:25:45.614465 13061 raft_consensus.cc:348] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } } peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } }
I20260923 17:25:45.614521 13060 consensus_queue.cc:260] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } } peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } }
I20260923 17:25:45.614816 13061 raft_consensus.cc:374] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:45.614920 13061 raft_consensus.cc:729] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f394a6ef0349e9b56fae2c8302c9de, State: Initialized, Role: FOLLOWER
I20260923 17:25:45.615811 12913 heartbeater.cc:503] Master 127.28.72.254:46851 was elected leader, sending a full tablet report...
I20260923 17:25:45.615688 13061 consensus_queue.cc:260] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } } peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } }
I20260923 17:25:45.617043 13060 ts_tablet_manager.cc:1428] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460: Time spent starting tablet: real 0.016s user 0.007s sys 0.009s
I20260923 17:25:45.618506 13061 ts_tablet_manager.cc:1428] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260923 17:25:45.619474 13062 raft_consensus.cc:348] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } } peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } }
I20260923 17:25:45.619863 13062 raft_consensus.cc:374] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:45.619992 13062 raft_consensus.cc:729] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d6ef19f6374f92932d361eb5c86dff, State: Initialized, Role: FOLLOWER
I20260923 17:25:45.620687 13062 consensus_queue.cc:260] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } } peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } }
I20260923 17:25:45.623152 13044 heartbeater.cc:503] Master 127.28.72.254:46851 was elected leader, sending a full tablet report...
I20260923 17:25:45.623906 13062 ts_tablet_manager.cc:1428] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
W20260923 17:25:45.704620 13045 tablet.cc:2369] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:25:45.720175 12914 tablet.cc:2369] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:25:45.725133 12782 tablet.cc:2369] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:25:45.822604 13066 raft_consensus.cc:482] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:25:45.822969 13066 raft_consensus.cc:504] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } } peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } }
I20260923 17:25:45.824877 13066 leader_election.cc:288] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d6ef19f6374f92932d361eb5c86dff (127.28.72.195:37251), 09f394a6ef0349e9b56fae2c8302c9de (127.28.72.193:39849)
I20260923 17:25:45.831760 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.825391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:56929 (local address 127.28.72.195:37251)
0923 17:25:45.825607 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:45.825614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:45.826436 (+ 822us) server_negotiation.cc:407] Connection header received
0923 17:25:45.826757 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:45.826761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:45.826825 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:45.826938 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:45.827488 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.828212 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.829974 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.830761 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.830965 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.831075 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:45.831358 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 17:25:45.831490 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 17:25:45.832085 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.825541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:37251 (local address 127.28.72.194:56929)
0923 17:25:45.826302 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:45.826320 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:45.826582 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:45.826945 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:45.826950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:45.826976 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:45.827339 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:45.827349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.828352 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.828357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:45.829804 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:45.829817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.830982 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.830988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:45.831194 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.831219 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:25:45.831330 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 17:25:45.831475 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":3,"thread_start_us":451,"threads_started":1}
I20260923 17:25:45.832962 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.825588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:39849 (local address 127.28.72.194:33545)
0923 17:25:45.826859 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:45.826875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:45.827057 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:45.827470 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:45.827474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:45.827500 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:45.827873 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:45.827883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.829015 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.829023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:45.830574 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:45.830587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.832048 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.832059 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:45.832338 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.832388 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 17:25:45.832497 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 17:25:45.832645 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"spinlock_wait_cycles":25472,"thread_start_us":943,"threads_started":1}
I20260923 17:25:45.832984 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.825392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:33545 (local address 127.28.72.193:39849)
0923 17:25:45.825608 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:45.825614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:45.826994 (+ 1380us) server_negotiation.cc:407] Connection header received
0923 17:25:45.827251 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:45.827257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:45.827337 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:45.827450 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:45.828053 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.828867 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.830752 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.831689 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.831914 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.832044 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:45.832579 (+ 535us) server_negotiation.cc:299] Negotiation successful
0923 17:25:45.832731 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:25:45.833252 12999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca0c64c70a24c92b70ef5d6e55c32df" candidate_uuid: "89d0fd247bf44f9fba3197863d854460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d6ef19f6374f92932d361eb5c86dff" is_pre_election: true
I20260923 17:25:45.833889 12999 raft_consensus.cc:2454] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89d0fd247bf44f9fba3197863d854460 in term 0.
I20260923 17:25:45.834110 12736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca0c64c70a24c92b70ef5d6e55c32df" candidate_uuid: "89d0fd247bf44f9fba3197863d854460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f394a6ef0349e9b56fae2c8302c9de" is_pre_election: true
I20260923 17:25:45.834745 12736 raft_consensus.cc:2454] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89d0fd247bf44f9fba3197863d854460 in term 0.
I20260923 17:25:45.834846 12801 leader_election.cc:302] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d6ef19f6374f92932d361eb5c86dff, 89d0fd247bf44f9fba3197863d854460; no voters:
I20260923 17:25:45.835346 13066 raft_consensus.cc:2781] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:25:45.835546 13066 raft_consensus.cc:482] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:25:45.835697 13066 raft_consensus.cc:3037] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:45.838030 13066 raft_consensus.cc:504] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } } peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } }
I20260923 17:25:45.838654 13066 leader_election.cc:288] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [CANDIDATE]: Term 1 election: Requested vote from peers 71d6ef19f6374f92932d361eb5c86dff (127.28.72.195:37251), 09f394a6ef0349e9b56fae2c8302c9de (127.28.72.193:39849)
I20260923 17:25:45.839305 12999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca0c64c70a24c92b70ef5d6e55c32df" candidate_uuid: "89d0fd247bf44f9fba3197863d854460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d6ef19f6374f92932d361eb5c86dff"
I20260923 17:25:45.839308 12736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca0c64c70a24c92b70ef5d6e55c32df" candidate_uuid: "89d0fd247bf44f9fba3197863d854460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f394a6ef0349e9b56fae2c8302c9de"
I20260923 17:25:45.839541 12999 raft_consensus.cc:3037] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:45.839541 12736 raft_consensus.cc:3037] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:45.842861 12736 raft_consensus.cc:2454] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89d0fd247bf44f9fba3197863d854460 in term 1.
I20260923 17:25:45.842861 12999 raft_consensus.cc:2454] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89d0fd247bf44f9fba3197863d854460 in term 1.
I20260923 17:25:45.843438 12801 leader_election.cc:302] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09f394a6ef0349e9b56fae2c8302c9de, 71d6ef19f6374f92932d361eb5c86dff, 89d0fd247bf44f9fba3197863d854460; no voters:
I20260923 17:25:45.843786 13066 raft_consensus.cc:2781] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:45.844390 13066 raft_consensus.cc:686] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [term 1 LEADER]: Becoming Leader. State: Replica: 89d0fd247bf44f9fba3197863d854460, State: Running, Role: LEADER
I20260923 17:25:45.845021 13066 consensus_queue.cc:237] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } } peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } }
I20260923 17:25:45.850284 12596 catalog_manager.cc:5495] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89d0fd247bf44f9fba3197863d854460 (127.28.72.194). New cstate: current_term: 1 leader_uuid: "89d0fd247bf44f9fba3197863d854460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89d0fd247bf44f9fba3197863d854460" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44777 } health_report { overall_health: HEALTHY } } }
I20260923 17:25:45.928992 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.923160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.28.72.194:44777)
0923 17:25:45.923380 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:45.923387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:45.923527 (+ 140us) server_negotiation.cc:407] Connection header received
0923 17:25:45.923789 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:45.923794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:45.923893 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:45.924050 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:45.924466 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.925181 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.926670 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:45.927102 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:45.927298 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:45.927845 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:45.928371 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:45.928534 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:45.928613 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 17:25:45.928699 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:25:45.937676 12736 raft_consensus.cc:1261] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de [term 1 FOLLOWER]: Refusing update from remote peer 89d0fd247bf44f9fba3197863d854460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:45.937764 12999 raft_consensus.cc:1261] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff [term 1 FOLLOWER]: Refusing update from remote peer 89d0fd247bf44f9fba3197863d854460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:45.938789 13071 consensus_queue.cc:1035] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d6ef19f6374f92932d361eb5c86dff" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 37251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:45.939174 13066 consensus_queue.cc:1035] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f394a6ef0349e9b56fae2c8302c9de" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:45.968695 12595 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:45.969074 12595 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:45.973812 12595 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Sending DeleteTablet for 3 replicas of tablet cca0c64c70a24c92b70ef5d6e55c32df
I20260923 17:25:45.975270 12716 tablet_service.cc:1514] Processing DeleteTablet for tablet cca0c64c70a24c92b70ef5d6e55c32df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:45 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:37782
I20260923 17:25:45.975857 12848 tablet_service.cc:1514] Processing DeleteTablet for tablet cca0c64c70a24c92b70ef5d6e55c32df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:45 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:57316
I20260923 17:25:45.976080 12979 tablet_service.cc:1514] Processing DeleteTablet for tablet cca0c64c70a24c92b70ef5d6e55c32df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:45 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:44408
I20260923 17:25:45.979455 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 12633
I20260923 17:25:45.981935 13085 tablet_replica.cc:330] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff: stopping tablet replica
I20260923 17:25:45.989669 13085 raft_consensus.cc:2229] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:45.990371 13085 raft_consensus.cc:2258] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:45.993264 13085 ts_tablet_manager.cc:1905] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:46.003221 13084 tablet_replica.cc:330] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460: stopping tablet replica
I20260923 17:25:46.004487 13083 tablet_replica.cc:330] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de: stopping tablet replica
I20260923 17:25:46.004917 13085 ts_tablet_manager.cc:1918] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:46.005172 13085 log.cc:1196] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/cca0c64c70a24c92b70ef5d6e55c32df
I20260923 17:25:46.005878 13085 ts_tablet_manager.cc:1939] T cca0c64c70a24c92b70ef5d6e55c32df P 71d6ef19f6374f92932d361eb5c86dff: Deleting consensus metadata
I20260923 17:25:46.007942 12580 catalog_manager.cc:4841] TS 71d6ef19f6374f92932d361eb5c86dff (127.28.72.195:37251): tablet cca0c64c70a24c92b70ef5d6e55c32df (table test-table [id=eda56ae165c74b119c89b8f5cfbafa73]) successfully deleted
I20260923 17:25:46.008570 13083 raft_consensus.cc:2229] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:46.009291 13083 raft_consensus.cc:2258] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:46.003799 13084 raft_consensus.cc:2229] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:25:46.010205 13084 raft_consensus.cc:2258] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:46.012748 13084 ts_tablet_manager.cc:1905] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:46.012768 13083 ts_tablet_manager.cc:1905] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:46.020669 13084 ts_tablet_manager.cc:1918] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:46.020762 13083 ts_tablet_manager.cc:1918] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:46.020946 13083 log.cc:1196] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/cca0c64c70a24c92b70ef5d6e55c32df
I20260923 17:25:46.021070 13084 log.cc:1196] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/cca0c64c70a24c92b70ef5d6e55c32df
I20260923 17:25:46.021597 13083 ts_tablet_manager.cc:1939] T cca0c64c70a24c92b70ef5d6e55c32df P 09f394a6ef0349e9b56fae2c8302c9de: Deleting consensus metadata
I20260923 17:25:46.021760 13084 ts_tablet_manager.cc:1939] T cca0c64c70a24c92b70ef5d6e55c32df P 89d0fd247bf44f9fba3197863d854460: Deleting consensus metadata
I20260923 17:25:46.022603 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:45.980764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.28.72.193:39849)
0923 17:25:45.981002 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:45.981009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:45.999982 (+ 18973us) server_negotiation.cc:407] Connection header received
0923 17:25:46.000305 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:46.000312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:46.000406 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:46.000491 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:46.005164 (+ 4673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:46.006395 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:46.008546 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:46.009253 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:46.009372 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:46.013582 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:46.013597 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:46.013614 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:46.013656 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:46.018465 (+ 4809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:46.019597 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:46.019608 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:46.019611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:46.019786 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:46.020299 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:46.020304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:46.020308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:46.020873 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:46.021221 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:46.021888 (+ 667us) server_negotiation.cc:299] Negotiation successful
0923 17:25:46.022208 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:25:46.023723 12583 catalog_manager.cc:4841] TS 09f394a6ef0349e9b56fae2c8302c9de (127.28.72.193:39849): tablet cca0c64c70a24c92b70ef5d6e55c32df (table test-table [id=eda56ae165c74b119c89b8f5cfbafa73]) successfully deleted
I20260923 17:25:46.023908 12582 catalog_manager.cc:4841] TS 89d0fd247bf44f9fba3197863d854460 (127.28.72.194:44777): tablet cca0c64c70a24c92b70ef5d6e55c32df (table test-table [id=eda56ae165c74b119c89b8f5cfbafa73]) successfully deleted
I20260923 17:25:46.024700 12776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:46.117146 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 12633
I20260923 17:25:46.143059 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 12785
I20260923 17:25:46.159901 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:46.143945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.28.72.194:44777)
0923 17:25:46.144125 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:46.144129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:46.144237 (+ 108us) server_negotiation.cc:407] Connection header received
0923 17:25:46.144471 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:46.144482 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:46.144572 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:46.144663 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:46.146560 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:46.147679 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:46.149657 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:46.150321 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:46.150535 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:46.153405 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:46.153428 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:46.153445 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:46.153486 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:46.156720 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:46.157558 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:46.157565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:46.157568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:46.157720 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:46.158128 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:46.158133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:46.158136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:46.158548 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:46.158895 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:46.159239 (+ 344us) server_negotiation.cc:299] Negotiation successful
0923 17:25:46.159496 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:25:46.160691 12908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:46.276877 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 12785
I20260923 17:25:46.296439 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 12916
I20260923 17:25:46.314459 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:46.297269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35400 (local address 127.28.72.195:37251)
0923 17:25:46.297520 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:46.297525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:46.297543 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:25:46.297615 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:46.297619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:46.297682 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:46.297763 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:46.299187 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:46.299914 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:46.301259 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:46.301644 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:46.301739 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:46.308716 (+ 6977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:46.308732 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:46.308749 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:46.308793 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:46.311613 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:46.312373 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:46.312381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:46.312383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:46.312516 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:46.312842 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:46.312845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:46.312847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:46.313210 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:46.313428 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:46.313812 (+ 384us) server_negotiation.cc:299] Negotiation successful
0923 17:25:46.314009 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260923 17:25:46.315372 13039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:46.407373 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 12916
I20260923 17:25:46.425267 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 12563
I20260923 17:25:46.426478 12626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:25:46.544103 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 12563
I20260923 17:26:06.567077 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:46851
--webserver_interface=127.28.72.254
--webserver_port=34121
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:46851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:06.914593 13091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:06.914966 13091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:06.915194 13091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:06.922974 13091 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:26:06.923156 13091 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:26:06.923260 13091 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:26:06.923359 13091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:26:06.923457 13091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:06.923555 13091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:06.923643 13091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:26:06.923719 13091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:26:06.934724 13091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:46851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:46851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=34121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:06.936190 13091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:06.937639 13091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:06.950312 13096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:06.950273 13099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:06.950273 13097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:06.951234 13091 server_base.cc:1034] running on GCE node
I20260923 17:26:06.952136 13091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:06.953747 13091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:06.955053 13091 hybrid_clock.cc:648] HybridClock initialized: now 1790184366954962 us; error 69 us; skew 500 ppm
I20260923 17:26:06.955554 13091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:06 dist-test-slave-j0m4 krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184366, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:06.965299 13091 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:26:06.967101 13091 webserver.cc:460] Webserver started at http://127.28.72.254:34121/ using document root <none> and password file <none>
I20260923 17:26:06.967927 13091 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:06.968026 13091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:06.974344 13091 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.005s
I20260923 17:26:06.977820 13106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:06.979580 13091 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260923 17:26:06.979780 13091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "4b9f217d69754c9f94cafce2891ef076"
format_stamp: "Formatted at 2026-09-23 17:25:20 on dist-test-slave-j0m4"
server_key: "2f2b066c943275468cd21afad03932e2"
server_key_iv: "b76db5170747720f893057c6a727d816"
server_key_version: "encryptionkey@0"
I20260923 17:26:06.980829 13091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:06.995795 13091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:07.000460 13091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:07.000785 13091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:07.021153 13091 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:46851
I20260923 17:26:07.021183 13157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:46851 every 8 connection(s)
I20260923 17:26:07.022878 13091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:26:07.030423 13158 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:26:07.032924 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 13091
I20260923 17:26:07.033766 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:39849
--local_ip_for_outbound_sockets=127.28.72.193
--tserver_master_addrs=127.28.72.254:46851
--webserver_port=44037
--webserver_interface=127.28.72.193
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:26:07.034056 13158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Bootstrap starting.
I20260923 17:26:07.055375 13158 log.cc:824] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:07.071568 13158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:26:07.072165 13158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Bootstrap complete.
I20260923 17:26:07.081743 13158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } }
I20260923 17:26:07.082863 13158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9f217d69754c9f94cafce2891ef076, State: Initialized, Role: FOLLOWER
I20260923 17:26:07.083554 13158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } }
I20260923 17:26:07.083819 13158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:26:07.083961 13158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:26:07.084139 13158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 2 FOLLOWER]: Advancing to term 3
I20260923 17:26:07.088594 13158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } }
I20260923 17:26:07.089174 13158 leader_election.cc:302] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b9f217d69754c9f94cafce2891ef076; no voters:
I20260923 17:26:07.089944 13158 leader_election.cc:288] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 17:26:07.090093 13162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 17:26:07.091535 13162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [term 3 LEADER]: Becoming Leader. State: Replica: 4b9f217d69754c9f94cafce2891ef076, State: Running, Role: LEADER
I20260923 17:26:07.092149 13162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } }
I20260923 17:26:07.092432 13158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:26:07.095083 13164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b9f217d69754c9f94cafce2891ef076. Latest consensus state: current_term: 3 leader_uuid: "4b9f217d69754c9f94cafce2891ef076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } } }
I20260923 17:26:07.095072 13163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4b9f217d69754c9f94cafce2891ef076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f217d69754c9f94cafce2891ef076" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 46851 } } }
I20260923 17:26:07.095577 13163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:07.095552 13164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:07.100454 13167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:26:07.116520 13167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:26:07.118273 13167 catalog_manager.cc:1259] Loaded cluster ID: efb6904c87e249dc996d0839a97fec2c
I20260923 17:26:07.118399 13167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:26:07.122834 13167 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:26:07.124760 13167 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Loaded TSK: 0
I20260923 17:26:07.124944 13167 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b9f217d69754c9f94cafce2891ef076: Loaded TSK: 1
F20260923 17:26:07.126142 13167 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f27cb6e39ad google::LogMessage::Fail() at ??:0
@ 0x7f27cb6e688a google::LogMessage::SendToLog() at ??:0
@ 0x7f27cb6e34de google::LogMessage::Flush() at ??:0
@ 0x7f27cb6e4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f27cf4477d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f27cf43673e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f27e11121c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f27e1111783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f27e119d2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f27e119d0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f27e11e75e9 std::function<>::operator()() at ??:0
@ 0x7f27e1115cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f27e1113c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f27e119b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f27e119b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f27ce32f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f27ce33291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f27ce332799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f27ce30deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f27d15e76db start_thread at ??:0
@ 0x7f27ca57d61f clone at ??:0
W20260923 17:26:07.466704 13160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:07.467001 13160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:07.467188 13160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:07.474841 13160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:07.474978 13160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:07.475236 13160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:26:07.485636 13160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:39849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=44037
--tserver_master_addrs=127.28.72.254:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:07.487044 13160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:07.488579 13160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:07.502583 13205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:07.502748 13206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:07.502920 13208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:07.503700 13160 server_base.cc:1034] running on GCE node
I20260923 17:26:07.504379 13160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:07.505405 13160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:07.506654 13160 hybrid_clock.cc:648] HybridClock initialized: now 1790184367506585 us; error 64 us; skew 500 ppm
I20260923 17:26:07.506997 13160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:07 dist-test-slave-j0m4 krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184367, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:07.510353 13160 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:26:07.512048 13160 webserver.cc:460] Webserver started at http://127.28.72.193:44037/ using document root <none> and password file <none>
I20260923 17:26:07.512758 13160 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:07.512869 13160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:07.518803 13160 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 17:26:07.521927 13215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:07.523541 13160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:26:07.523721 13160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "09f394a6ef0349e9b56fae2c8302c9de"
format_stamp: "Formatted at 2026-09-23 17:25:21 on dist-test-slave-j0m4"
server_key: "9ebaac2a4080c4e56b103b4876fe1f78"
server_key_iv: "cae692ef66ac42a81302654aea67a915"
server_key_version: "encryptionkey@0"
I20260923 17:26:07.524739 13160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:07.542703 13160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:07.547356 13160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:07.547660 13160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:07.548717 13160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:07.550208 13160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:07.550307 13160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:07.550427 13160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:07.550515 13160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:07.587066 13160 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:39849
I20260923 17:26:07.587172 13327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:39849 every 8 connection(s)
I20260923 17:26:07.588672 13160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:26:07.590279 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 13160
I20260923 17:26:07.590996 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:44777
--local_ip_for_outbound_sockets=127.28.72.194
--tserver_master_addrs=127.28.72.254:46851
--webserver_port=33629
--webserver_interface=127.28.72.194
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:26:07.599437 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:07.591991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:59031)
0923 17:26:07.597853 (+ 5862us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:07.598224 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5471,"thread_start_us":5217,"threads_started":1}
W20260923 17:26:07.600191 13328 heartbeater.cc:644] Failed to heartbeat to 127.28.72.254:46851 (0 consecutive failures): Network error: Failed to ping master at 127.28.72.254:46851: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
I20260923 17:26:07.601133 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:07.600780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:47729)
0923 17:26:07.600944 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:07.601027 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 17:26:07.601989 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:07.601657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:36889)
0923 17:26:07.601801 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:07.601883 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 17:26:07.602218 13328 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:26:07.938772 13331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:07.939124 13331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:07.939339 13331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:07.946789 13331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:07.946959 13331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:07.947177 13331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:26:07.957448 13331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:44777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=33629
--tserver_master_addrs=127.28.72.254:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:07.958855 13331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:07.960314 13331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:07.975679 13339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:07.975662 13336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:07.975752 13337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:07.976509 13331 server_base.cc:1034] running on GCE node
I20260923 17:26:07.977185 13331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:07.978248 13331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:07.979527 13331 hybrid_clock.cc:648] HybridClock initialized: now 1790184367979472 us; error 46 us; skew 500 ppm
I20260923 17:26:07.979895 13331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:07 dist-test-slave-j0m4 krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184367, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:07.983597 13331 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:26:07.985407 13331 webserver.cc:460] Webserver started at http://127.28.72.194:33629/ using document root <none> and password file <none>
I20260923 17:26:07.986198 13331 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:07.986371 13331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:07.992713 13331 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:26:07.995781 13346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:07.997346 13331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:26:07.997550 13331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "89d0fd247bf44f9fba3197863d854460"
format_stamp: "Formatted at 2026-09-23 17:25:21 on dist-test-slave-j0m4"
server_key: "48bac22c6e255f610380e9e158943c69"
server_key_iv: "6f9f31754f6c9089c6ea99987d3c2097"
server_key_version: "encryptionkey@0"
I20260923 17:26:07.998673 13331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:08.012521 13331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:08.017184 13331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:08.017505 13331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:08.018522 13331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:08.020008 13331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:08.020135 13331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:08.020272 13331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:08.020361 13331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:08.055320 13331 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:44777
I20260923 17:26:08.055414 13458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:44777 every 8 connection(s)
I20260923 17:26:08.056856 13331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:26:08.057355 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 13331
I20260923 17:26:08.058194 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:37251
--local_ip_for_outbound_sockets=127.28.72.195
--tserver_master_addrs=127.28.72.254:46851
--webserver_port=36601
--webserver_interface=127.28.72.195
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:26:08.062471 13461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:08.060350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:35945)
0923 17:26:08.061223 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:08.061489 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":358,"threads_started":1}
W20260923 17:26:08.063378 13459 heartbeater.cc:644] Failed to heartbeat to 127.28.72.254:46851 (0 consecutive failures): Network error: Failed to ping master at 127.28.72.254:46851: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
I20260923 17:26:08.064375 13461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:08.064000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:45765)
0923 17:26:08.064170 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:08.064260 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 17:26:08.065402 13461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:08.064956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:47087)
0923 17:26:08.065137 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:08.065241 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 17:26:08.065711 13459 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:26:08.397137 13462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:08.397441 13462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:08.397547 13462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:08.404927 13462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:08.405041 13462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:08.405156 13462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:26:08.415149 13462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:37251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=36601
--tserver_master_addrs=127.28.72.254:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:08.416464 13462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:08.417759 13462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:08.431401 13468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:08.431563 13470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:08.431631 13467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:08.432228 13462 server_base.cc:1034] running on GCE node
I20260923 17:26:08.432884 13462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:08.433983 13462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:08.435240 13462 hybrid_clock.cc:648] HybridClock initialized: now 1790184368435108 us; error 103 us; skew 500 ppm
I20260923 17:26:08.435582 13462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:08 dist-test-slave-j0m4 krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184368, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:08.438776 13462 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:26:08.440415 13462 webserver.cc:460] Webserver started at http://127.28.72.195:36601/ using document root <none> and password file <none>
I20260923 17:26:08.441136 13462 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:08.441285 13462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:08.446810 13462 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:26:08.449788 13477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:08.451509 13462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:26:08.451714 13462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "71d6ef19f6374f92932d361eb5c86dff"
format_stamp: "Formatted at 2026-09-23 17:25:22 on dist-test-slave-j0m4"
server_key: "c26f536d64de38265a2c14cc3fd576b6"
server_key_iv: "1afd5340d6bc5d8eb40845c5c887c9bd"
server_key_version: "encryptionkey@0"
I20260923 17:26:08.452819 13462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:08.475023 13462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:08.479615 13462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:08.479930 13462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:08.480980 13462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:08.482391 13462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:08.482527 13462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:08.482672 13462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:08.482762 13462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:08.517490 13462 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:37251
I20260923 17:26:08.517577 13589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:37251 every 8 connection(s)
I20260923 17:26:08.518982 13462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:26:08.523911 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:08.521995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:48961)
0923 17:26:08.522738 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:08.522973 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":290,"threads_started":1}
W20260923 17:26:08.524745 13590 heartbeater.cc:644] Failed to heartbeat to 127.28.72.254:46851 (0 consecutive failures): Network error: Failed to ping master at 127.28.72.254:46851: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
I20260923 17:26:08.525372 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 13462
I20260923 17:26:08.525679 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:08.525342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:48223)
0923 17:26:08.525503 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:08.525582 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 17:26:08.526616 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:08.526236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:34957)
0923 17:26:08.526414 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:08.526499 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260923 17:26:08.526871 13590 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:26:08.603858 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:08.602834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:59453)
0923 17:26:08.603425 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:08.603569 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":227,"threads_started":1}
I20260923 17:26:09.067579 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:09.066375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:39207)
0923 17:26:09.067042 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:09.067225 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":387,"threads_started":1}
I20260923 17:26:09.528452 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:09.527544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:48027)
0923 17:26:09.528092 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:09.528202 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":238,"threads_started":1}
I20260923 17:26:09.606316 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:09.605261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:34001)
0923 17:26:09.606007 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:09.606122 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":340,"threads_started":1}
I20260923 17:26:10.069677 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:10.068779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:46673)
0923 17:26:10.069404 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:10.069501 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":275,"threads_started":1}
I20260923 17:26:10.530332 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:10.529490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:38577)
0923 17:26:10.530070 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:10.530153 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":325,"threads_started":1}
I20260923 17:26:10.608268 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:10.607497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:55951)
0923 17:26:10.607982 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:10.608074 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":248,"threads_started":1}
I20260923 17:26:11.071656 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:11.070803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:55703)
0923 17:26:11.071383 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:11.071482 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":349,"threads_started":1}
I20260923 17:26:11.532315 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:11.531557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:48681)
0923 17:26:11.532049 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:11.532137 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":245,"threads_started":1}
I20260923 17:26:11.610085 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:11.609353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:33801)
0923 17:26:11.609812 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:11.609901 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":229,"threads_started":1}
I20260923 17:26:12.073369 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:12.072604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:40147)
0923 17:26:12.073069 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:12.073159 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":228,"threads_started":1}
I20260923 17:26:12.534165 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:12.533350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:42037)
0923 17:26:12.533876 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:12.533971 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":264,"threads_started":1}
I20260923 17:26:12.612084 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:12.611297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:54127)
0923 17:26:12.611804 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:12.611898 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":255,"threads_started":1}
I20260923 17:26:13.075330 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:13.074527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:52789)
0923 17:26:13.075050 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:13.075153 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":254,"threads_started":1}
I20260923 17:26:13.536193 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:13.535379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:46771)
0923 17:26:13.535913 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:13.536020 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":268,"threads_started":1}
I20260923 17:26:13.614213 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:13.613424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:55419)
0923 17:26:13.613939 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:13.614043 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":255,"threads_started":1}
I20260923 17:26:14.077117 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:14.076373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:34307)
0923 17:26:14.076849 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:14.076950 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":245,"threads_started":1}
I20260923 17:26:14.538101 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:14.537207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:40003)
0923 17:26:14.537800 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:14.537934 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":316,"threads_started":1}
I20260923 17:26:14.616345 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:14.615442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:37639)
0923 17:26:14.616014 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:14.616108 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":277,"threads_started":1}
I20260923 17:26:15.079360 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:15.078269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:42375)
0923 17:26:15.078923 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:15.079021 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":307,"threads_started":1}
I20260923 17:26:15.540361 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:15.539291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:48057)
0923 17:26:15.539977 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:15.540076 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":321,"threads_started":1}
I20260923 17:26:15.618283 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:15.617495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:44653)
0923 17:26:15.618007 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:15.618102 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":262,"threads_started":1}
I20260923 17:26:16.081380 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:16.080512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:48509)
0923 17:26:16.081046 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:16.081141 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":275,"threads_started":1}
I20260923 17:26:16.542354 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:16.541537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:44699)
0923 17:26:16.542088 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:16.542185 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":276,"threads_started":1}
I20260923 17:26:16.620553 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:16.619675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:48135)
0923 17:26:16.620264 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:16.620370 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":311,"threads_started":1}
I20260923 17:26:17.083465 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:17.082500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:40929)
0923 17:26:17.083063 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:17.083253 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":289,"threads_started":1}
I20260923 17:26:17.544220 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:17.543408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:44297)
0923 17:26:17.543937 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:17.544025 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":262,"threads_started":1}
I20260923 17:26:17.622556 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:17.621766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:36739)
0923 17:26:17.622308 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:17.622386 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":303,"threads_started":1}
I20260923 17:26:18.085435 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:18.084656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:47145)
0923 17:26:18.085165 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:18.085257 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":283,"threads_started":1}
I20260923 17:26:18.546232 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:18.545487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:41555)
0923 17:26:18.545985 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:18.546071 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":262,"threads_started":1}
I20260923 17:26:18.624364 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:18.623591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:35213)
0923 17:26:18.624081 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:18.624178 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":251,"threads_started":1}
I20260923 17:26:19.087633 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:19.086718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:46377)
0923 17:26:19.087341 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:19.087447 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":312,"threads_started":1}
I20260923 17:26:19.548223 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:19.547420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:58575)
0923 17:26:19.547924 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:19.548036 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":261,"threads_started":1}
I20260923 17:26:19.626264 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:19.625494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:45419)
0923 17:26:19.625994 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:19.626088 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":249,"threads_started":1}
I20260923 17:26:20.089412 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:20.088673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:33391)
0923 17:26:20.089148 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:20.089242 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":273,"threads_started":1}
I20260923 17:26:20.550143 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:20.549344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:60845)
0923 17:26:20.549875 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:20.549973 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
I20260923 17:26:20.628543 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:20.627529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:60273)
0923 17:26:20.628163 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:20.628274 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":329,"threads_started":1}
I20260923 17:26:21.091463 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:21.090537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:42945)
0923 17:26:21.091062 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:21.091211 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":262,"threads_started":1}
I20260923 17:26:21.552240 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:21.551384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:43241)
0923 17:26:21.551943 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:21.552047 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":270,"threads_started":1}
I20260923 17:26:21.630419 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:21.629568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:34983)
0923 17:26:21.630068 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:21.630160 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":276,"threads_started":1}
I20260923 17:26:22.093192 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.092327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:58273)
0923 17:26:22.092892 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:22.092991 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":254,"threads_started":1}
I20260923 17:26:22.554400 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.553305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:42833)
0923 17:26:22.553855 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:22.554144 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":292,"threads_started":1}
I20260923 17:26:22.632638 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.631725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.193:35713)
0923 17:26:22.632212 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:22.632417 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260923 17:26:23.095764 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:23.094728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.194:34073)
0923 17:26:23.095372 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:23.095474 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":342,"threads_started":1}
I20260923 17:26:23.539330 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 13160
I20260923 17:26:23.556556 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:23.555669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:46851 (local address 127.28.72.195:56101)
0923 17:26:23.556205 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:23.556329 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.254:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":281,"threads_started":1}
I20260923 17:26:23.558507 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:23.540709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.28.72.193:39849)
0923 17:26:23.541334 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:23.541351 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:23.541396 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:26:23.542953 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:23.542964 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:23.543525 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:23.543834 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:23.545321 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:23.546618 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:23.547929 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:23.548624 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:23.548770 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:23.551543 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:23.551551 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:23.551567 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:23.551610 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:23.554733 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:23.555563 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:23.555571 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:23.555578 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:23.555770 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:23.556235 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:23.556241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:23.556243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:23.556979 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:23.557304 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:23.557762 (+ 458us) server_negotiation.cc:299] Negotiation successful
0923 17:26:23.558224 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":234,"threads_started":1}
I20260923 17:26:23.560227 13323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:23.645622 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 13160
I20260923 17:26:23.664129 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 13331
I20260923 17:26:23.682175 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:23.665119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.28.72.194:44777)
0923 17:26:23.665675 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:23.665689 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:23.665727 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:26:23.667362 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:23.667372 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:23.667816 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:23.668116 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:23.669647 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:23.670872 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:23.672299 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:23.672982 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:23.673126 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:23.675897 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:23.675905 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:23.675921 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:23.675965 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:23.678999 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:23.679777 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:23.679783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:23.679790 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:23.679966 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:23.680307 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:23.680310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:23.680312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:23.680879 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:23.681113 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:23.681550 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 17:26:23.681890 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":242,"threads_started":1}
I20260923 17:26:23.683750 13454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:23.769099 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 13331
I20260923 17:26:23.787556 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 13462
I20260923 17:26:23.807471 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:23.788633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40532 (local address 127.28.72.195:37251)
0923 17:26:23.789207 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:23.789221 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:23.789262 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:26:23.790966 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:23.790977 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:23.791531 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:23.791954 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:23.793478 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:23.794717 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:23.796275 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:23.797000 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:23.797226 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:23.800129 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:23.800138 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:23.800155 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:23.800198 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:23.803784 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:23.804610 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:23.804618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:23.804625 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:23.804819 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:23.805246 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:23.805253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:23.805260 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:23.805845 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:23.806196 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:23.806626 (+ 430us) server_negotiation.cc:299] Negotiation successful
0923 17:26:23.807091 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":274,"threads_started":1}
I20260923 17:26:23.808914 13585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:23.896646 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 13462
2026-09-23T17:26:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64976 ms)
[----------] 30 tests from SecurityITest (313333 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:26:24 dist-test-slave-j0m4 krb5kdc[13648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:26:24 dist-test-slave-j0m4 krb5kdc[13648](info): set up 2 sockets
Sep 23 17:26:24 dist-test-slave-j0m4 krb5kdc[13648](info): commencing operation
krb5kdc: starting...
W20260923 17:26:25.019767 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:26:25 dist-test-slave-j0m4 krb5kdc[13648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:26:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:26:25.295194 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:45801
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:45801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:25.647207 13661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:25.647542 13661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:25.647709 13661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:25.655495 13661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:26:25.655618 13661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:25.655691 13661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:25.655781 13661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:26:25.655872 13661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:26:25.666152 13661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:45801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:45801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:25.667647 13661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:25.669064 13661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:25.681438 13666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:25.681519 13667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:25.681578 13669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:25.682410 13661 server_base.cc:1034] running on GCE node
I20260923 17:26:25.683308 13661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:25.684880 13661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:25.686132 13661 hybrid_clock.cc:648] HybridClock initialized: now 1790184385686078 us; error 46 us; skew 500 ppm
I20260923 17:26:25.686564 13661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:25 dist-test-slave-j0m4 krb5kdc[13648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184385, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:25.690167 13661 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:26:25.692065 13661 webserver.cc:460] Webserver started at http://127.28.72.254:33655/ using document root <none> and password file <none>
I20260923 17:26:25.692826 13661 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:25.692987 13661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:25.693320 13661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:25.696681 13661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "548461735def4164969d00e31f7cfc3c"
format_stamp: "Formatted at 2026-09-23 17:26:25 on dist-test-slave-j0m4"
server_key: "60b57c14b0e0c8a8df73937e23d55ac9"
server_key_iv: "c1f5374be9f0a21578242b57e03ac1b8"
server_key_version: "encryptionkey@0"
I20260923 17:26:25.697600 13661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "548461735def4164969d00e31f7cfc3c"
format_stamp: "Formatted at 2026-09-23 17:26:25 on dist-test-slave-j0m4"
server_key: "60b57c14b0e0c8a8df73937e23d55ac9"
server_key_iv: "c1f5374be9f0a21578242b57e03ac1b8"
server_key_version: "encryptionkey@0"
I20260923 17:26:25.703503 13661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:26:25.707662 13676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:25.709239 13661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:26:25.709465 13661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "548461735def4164969d00e31f7cfc3c"
format_stamp: "Formatted at 2026-09-23 17:26:25 on dist-test-slave-j0m4"
server_key: "60b57c14b0e0c8a8df73937e23d55ac9"
server_key_iv: "c1f5374be9f0a21578242b57e03ac1b8"
server_key_version: "encryptionkey@0"
I20260923 17:26:25.709693 13661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:25.735500 13661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:25.740396 13661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:25.740722 13661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:25.761137 13661 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:45801
I20260923 17:26:25.761168 13727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:45801 every 8 connection(s)
I20260923 17:26:25.762555 13661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:26:25.766444 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:25.772521 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 13661
I20260923 17:26:25.772758 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:26:25.773594 28963 external_mini_cluster.cc:1428] Setting key 4a9f563e9acae282f559b95409ff70e3
I20260923 17:26:25.778492 13728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c: Bootstrap starting.
I20260923 17:26:25.782516 13728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:25.783845 13728 log.cc:824] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:25.787503 13728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c: No bootstrap required, opened a new log
Sep 23 17:26:25 dist-test-slave-j0m4 krb5kdc[13648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:26:25.796702 13728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548461735def4164969d00e31f7cfc3c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45801 } }
I20260923 17:26:25.797187 13728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:25.797338 13728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548461735def4164969d00e31f7cfc3c, State: Initialized, Role: FOLLOWER
I20260923 17:26:25.798066 13728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548461735def4164969d00e31f7cfc3c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45801 } }
I20260923 17:26:25.798354 13728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:26:25.798566 13728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:26:25.798825 13728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:25.800244 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:25.775996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.28.72.254:45801)
0923 17:26:25.776850 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:25.776862 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:25.776904 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:26:25.778681 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:25.778720 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:25.779201 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:25.779556 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:25.781502 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:25.782981 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:25.784961 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:25.785802 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:25.785941 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:25.791108 (+ 5167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:25.791168 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:25.791197 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:25.791261 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:25.794977 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:25.796036 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:25.796045 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:25.796052 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:25.796261 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:25.796744 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:25.796749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:25.796751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:25.797508 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:25.797852 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:25.798378 (+ 526us) server_negotiation.cc:299] Negotiation successful
0923 17:26:25.798866 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":413,"threads_started":1}
I20260923 17:26:25.802191 13728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548461735def4164969d00e31f7cfc3c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45801 } }
I20260923 17:26:25.802765 13728 leader_election.cc:302] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 548461735def4164969d00e31f7cfc3c; no voters:
I20260923 17:26:25.804302 13728 leader_election.cc:288] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:26:25.804601 13733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:25.805727 13733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [term 1 LEADER]: Becoming Leader. State: Replica: 548461735def4164969d00e31f7cfc3c, State: Running, Role: LEADER
I20260923 17:26:25.806432 13733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548461735def4164969d00e31f7cfc3c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45801 } }
I20260923 17:26:25.806838 13728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:26:25.809825 13734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "548461735def4164969d00e31f7cfc3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548461735def4164969d00e31f7cfc3c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45801 } } }
I20260923 17:26:25.810002 13735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 548461735def4164969d00e31f7cfc3c. Latest consensus state: current_term: 1 leader_uuid: "548461735def4164969d00e31f7cfc3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548461735def4164969d00e31f7cfc3c" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45801 } } }
I20260923 17:26:25.812393 13734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:25.812397 13735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:25.819936 13744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:26:25.825654 13744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:26:25.834918 13744 catalog_manager.cc:1347] Generated new cluster ID: 9019bcb559b34434b3f0576818f515bf
I20260923 17:26:25.835058 13744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:26:25.870457 13744 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:26:25.872519 13744 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:26:25.883025 13744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c: Generated new TSK 0
I20260923 17:26:25.884192 13744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:26:25.960934 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:45801
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:26.322278 13754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:26.322599 13754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:26.322758 13754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:26.330435 13754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:26.330570 13754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:26.330747 13754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:26:26.340893 13754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:45801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:26.342381 13754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:26.343813 13754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:26.358367 13760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:26.358407 13759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:26.358450 13762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:26.359220 13754 server_base.cc:1034] running on GCE node
I20260923 17:26:26.359851 13754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:26.360939 13754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:26.362151 13754 hybrid_clock.cc:648] HybridClock initialized: now 1790184386362084 us; error 62 us; skew 500 ppm
I20260923 17:26:26.362461 13754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:26 dist-test-slave-j0m4 krb5kdc[13648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184386, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:26.366045 13754 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:26:26.367843 13754 webserver.cc:460] Webserver started at http://127.28.72.193:40399/ using document root <none> and password file <none>
I20260923 17:26:26.368551 13754 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:26.368657 13754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:26.368932 13754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:26.372078 13754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "e8e7771d26124e639ac4e3f095a83a6d"
format_stamp: "Formatted at 2026-09-23 17:26:26 on dist-test-slave-j0m4"
server_key: "f8c07ac0e46886bdcafd2add8da4398e"
server_key_iv: "22208cb6598a0b2dcdf83d4abc4f4286"
server_key_version: "encryptionkey@0"
I20260923 17:26:26.372849 13754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "e8e7771d26124e639ac4e3f095a83a6d"
format_stamp: "Formatted at 2026-09-23 17:26:26 on dist-test-slave-j0m4"
server_key: "f8c07ac0e46886bdcafd2add8da4398e"
server_key_iv: "22208cb6598a0b2dcdf83d4abc4f4286"
server_key_version: "encryptionkey@0"
I20260923 17:26:26.378654 13754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:26:26.382460 13769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:26.384114 13754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:26:26.384280 13754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "e8e7771d26124e639ac4e3f095a83a6d"
format_stamp: "Formatted at 2026-09-23 17:26:26 on dist-test-slave-j0m4"
server_key: "f8c07ac0e46886bdcafd2add8da4398e"
server_key_iv: "22208cb6598a0b2dcdf83d4abc4f4286"
server_key_version: "encryptionkey@0"
I20260923 17:26:26.384481 13754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:26.415725 13754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:26.420171 13754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:26.420436 13754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:26.421401 13754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:26.422885 13754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:26.423045 13754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:26.423250 13754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:26.423346 13754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:26.457963 13754 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:34033
I20260923 17:26:26.458007 13881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:34033 every 8 connection(s)
I20260923 17:26:26.459409 13754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:26:26.468420 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 13754
I20260923 17:26:26.468644 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:26:26.469293 28963 external_mini_cluster.cc:1428] Setting key d2ea50eace42ac97e0d700f7a78e13a4
Sep 23 17:26:26 dist-test-slave-j0m4 krb5kdc[13648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184386, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:26:26.486811 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:26.462733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:49915 (local address 127.28.72.254:45801)
0923 17:26:26.463382 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:26.463387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:26.464385 (+ 998us) server_negotiation.cc:407] Connection header received
0923 17:26:26.466357 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:26.466362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:26.466457 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:26.466579 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:26.468249 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:26.469117 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:26.471262 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:26.471841 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:26.472000 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:26.480258 (+ 8258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:26.480264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:26.480268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:26.480315 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:26.483163 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:26.484305 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:26.484310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:26.484312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:26.484442 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:26.484976 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:26.484979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:26.484981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:26.485243 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:26.485354 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:26.486337 (+ 983us) server_negotiation.cc:299] Negotiation successful
0923 17:26:26.486534 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":260,"threads_started":1}
I20260923 17:26:26.488570 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:26.463392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:45801 (local address 127.28.72.193:49915)
0923 17:26:26.464200 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:26.464243 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:26.466056 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:26.466836 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:26.466845 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:26.467461 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:26.468070 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:26.468088 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:26.469284 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:26.469288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:26.471062 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:26.471078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:26.472089 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:26.472094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:26.472315 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:26.476875 (+ 4560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:26:26.476931 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:26.479917 (+ 2986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:26.483374 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:26.483389 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:26.483413 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:26.484156 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:26.484558 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:26.484566 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:26.484572 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:26.484857 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:26.485382 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:26.485394 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:26.485912 (+ 518us) client_negotiation.cc:769] Sending connection context
0923 17:26:26.486219 (+ 307us) client_negotiation.cc:240] Negotiation successful
0923 17:26:26.486865 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":350,"threads_started":1}
I20260923 17:26:26.490451 13882 heartbeater.cc:347] Connected to a master server at 127.28.72.254:45801
I20260923 17:26:26.490818 13882 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:26.491796 13882 heartbeater.cc:511] Master 127.28.72.254:45801 requested a full tablet report, sending...
I20260923 17:26:26.494207 13693 ts_manager.cc:194] Registered new tserver with Master: e8e7771d26124e639ac4e3f095a83a6d (127.28.72.193:34033)
I20260923 17:26:26.495882 13693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:49915
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:26:26.532069 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:45801
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:26.881389 13888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:26.881768 13888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:26.881943 13888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:26.890038 13888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:26.890215 13888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:26.890398 13888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:26:26.900789 13888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:45801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:26.902245 13888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:26.903714 13888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:26.918570 13894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:26.918610 13896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:26.918500 13893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:26.919719 13888 server_base.cc:1034] running on GCE node
I20260923 17:26:26.920424 13888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:26.921487 13888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:26.922762 13888 hybrid_clock.cc:648] HybridClock initialized: now 1790184386922676 us; error 53 us; skew 500 ppm
I20260923 17:26:26.923178 13888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:26 dist-test-slave-j0m4 krb5kdc[13648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184386, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:26.926630 13888 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:26:26.928846 13888 webserver.cc:460] Webserver started at http://127.28.72.194:36329/ using document root <none> and password file <none>
I20260923 17:26:26.929577 13888 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:26.929695 13888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:26.929978 13888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:26.933269 13888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "8fe1dd594a854b4ba87eb84f15ea7901"
format_stamp: "Formatted at 2026-09-23 17:26:26 on dist-test-slave-j0m4"
server_key: "4b9d058ee80738e599469d9ae6c3a2d6"
server_key_iv: "db00685330d1decd68595538fc71d865"
server_key_version: "encryptionkey@0"
I20260923 17:26:26.934127 13888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "8fe1dd594a854b4ba87eb84f15ea7901"
format_stamp: "Formatted at 2026-09-23 17:26:26 on dist-test-slave-j0m4"
server_key: "4b9d058ee80738e599469d9ae6c3a2d6"
server_key_iv: "db00685330d1decd68595538fc71d865"
server_key_version: "encryptionkey@0"
I20260923 17:26:26.940178 13888 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 17:26:26.944070 13903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:26.945720 13888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:26:26.945887 13888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "8fe1dd594a854b4ba87eb84f15ea7901"
format_stamp: "Formatted at 2026-09-23 17:26:26 on dist-test-slave-j0m4"
server_key: "4b9d058ee80738e599469d9ae6c3a2d6"
server_key_iv: "db00685330d1decd68595538fc71d865"
server_key_version: "encryptionkey@0"
I20260923 17:26:26.946094 13888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:26.971772 13888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:26.976768 13888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:26.977080 13888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:26.978346 13888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:26.979849 13888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:26.979966 13888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:26.980125 13888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:26.980199 13888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:27.016079 13888 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:38373
I20260923 17:26:27.016139 14015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:38373 every 8 connection(s)
I20260923 17:26:27.017468 13888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:26:27.018908 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 13888
I20260923 17:26:27.019141 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:26:27.019706 28963 external_mini_cluster.cc:1428] Setting key 61b72fa4c22d12cfb36cb7b0cce988fc
Sep 23 17:26:27 dist-test-slave-j0m4 krb5kdc[13648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184386, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:26:27.044808 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.020720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:45859 (local address 127.28.72.254:45801)
0923 17:26:27.021756 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:27.021763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:27.022668 (+ 905us) server_negotiation.cc:407] Connection header received
0923 17:26:27.025843 (+ 3175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:27.025848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:27.025925 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:27.026040 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:27.028493 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.029239 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.031965 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.032739 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.032949 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.038021 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:27.038032 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:27.038038 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:27.038092 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:27.041274 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:27.042140 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:27.042152 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:27.042154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:27.042286 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:27.042763 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:27.042771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:27.042779 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:27.043070 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:27.043313 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:27.044132 (+ 819us) server_negotiation.cc:299] Negotiation successful
0923 17:26:27.044398 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":797,"threads_started":1}
I20260923 17:26:27.045981 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.021101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:45801 (local address 127.28.72.194:45859)
0923 17:26:27.022464 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:27.022526 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:27.025455 (+ 2929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:27.026349 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:27.026365 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:27.027250 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:27.028209 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:27.028240 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.029486 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.029495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:27.031762 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:27.031782 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.032981 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.032986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:27.033194 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.034032 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:26:27.034068 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:27.037692 (+ 3624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:27.041459 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:27.041470 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:27.041489 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:27.041980 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:27.042411 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:27.042416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:27.042420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:27.042633 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:27.043282 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:27.043291 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:27.043704 (+ 413us) client_negotiation.cc:769] Sending connection context
0923 17:26:27.044027 (+ 323us) client_negotiation.cc:240] Negotiation successful
0923 17:26:27.044652 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":740,"threads_started":1}
I20260923 17:26:27.047683 14016 heartbeater.cc:347] Connected to a master server at 127.28.72.254:45801
I20260923 17:26:27.048031 14016 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
I20260923 17:26:27.048835 14016 heartbeater.cc:511] Master 127.28.72.254:45801 requested a full tablet report, sending...
I20260923 17:26:27.050573 13693 ts_manager.cc:194] Registered new tserver with Master: 8fe1dd594a854b4ba87eb84f15ea7901 (127.28.72.194:38373)
I20260923 17:26:27.051600 13693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:45859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:26:27.083943 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:45801
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:27.431308 14022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:27.431605 14022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:27.431771 14022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:27.439502 14022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:27.439617 14022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:27.439814 14022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:26:27.450408 14022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:45801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:27.452111 14022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:27.453723 14022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:27.468222 14027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:27.468303 14028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:27.468261 14030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:27.469200 14022 server_base.cc:1034] running on GCE node
I20260923 17:26:27.469900 14022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:27.470948 14022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:27.472218 14022 hybrid_clock.cc:648] HybridClock initialized: now 1790184387472158 us; error 56 us; skew 500 ppm
I20260923 17:26:27.472637 14022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:27 dist-test-slave-j0m4 krb5kdc[13648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184387, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:27.476298 14022 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:26:27.478075 14022 webserver.cc:460] Webserver started at http://127.28.72.195:44039/ using document root <none> and password file <none>
I20260923 17:26:27.478875 14022 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:27.479018 14022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:27.479480 14022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:27.482515 14022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "8c196ee2550a468ca08bd7d24100ac8b"
format_stamp: "Formatted at 2026-09-23 17:26:27 on dist-test-slave-j0m4"
server_key: "5b94017a2dfd6d46682d7a85ab3c2c0e"
server_key_iv: "059b751310e0db82fa664e222bf27d32"
server_key_version: "encryptionkey@0"
I20260923 17:26:27.483336 14022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "8c196ee2550a468ca08bd7d24100ac8b"
format_stamp: "Formatted at 2026-09-23 17:26:27 on dist-test-slave-j0m4"
server_key: "5b94017a2dfd6d46682d7a85ab3c2c0e"
server_key_iv: "059b751310e0db82fa664e222bf27d32"
server_key_version: "encryptionkey@0"
I20260923 17:26:27.488757 14022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260923 17:26:27.492547 14037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:27.493981 14022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:26:27.494164 14022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "8c196ee2550a468ca08bd7d24100ac8b"
format_stamp: "Formatted at 2026-09-23 17:26:27 on dist-test-slave-j0m4"
server_key: "5b94017a2dfd6d46682d7a85ab3c2c0e"
server_key_iv: "059b751310e0db82fa664e222bf27d32"
server_key_version: "encryptionkey@0"
I20260923 17:26:27.494385 14022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:27.499531 13882 heartbeater.cc:503] Master 127.28.72.254:45801 was elected leader, sending a full tablet report...
I20260923 17:26:27.525568 14022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:27.530277 14022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:27.530618 14022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:27.531734 14022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:27.533219 14022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:27.533324 14022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:27.533483 14022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:27.533566 14022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:27.569120 14022 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:36975
I20260923 17:26:27.569218 14149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:36975 every 8 connection(s)
I20260923 17:26:27.570580 14022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:26:27.571635 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 14022
I20260923 17:26:27.571830 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:26:27.572417 28963 external_mini_cluster.cc:1428] Setting key 71be2b5007d7476c420750af81160624
Sep 23 17:26:27 dist-test-slave-j0m4 krb5kdc[13648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184387, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:26:27.593919 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.573782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:34135 (local address 127.28.72.254:45801)
0923 17:26:27.574293 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:27.574298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:27.575928 (+ 1630us) server_negotiation.cc:407] Connection header received
0923 17:26:27.577892 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:27.577896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:27.577991 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:27.578090 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:27.579998 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.580787 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.582461 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.582851 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.582964 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.587038 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:27.587043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:27.587046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:27.587083 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:27.590614 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:27.591490 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:27.591495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:27.591501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:27.591630 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:27.592094 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:27.592103 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:27.592105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:27.592374 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:27.592550 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:27.593264 (+ 714us) server_negotiation.cc:299] Negotiation successful
0923 17:26:27.593504 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":266,"threads_started":1}
I20260923 17:26:27.595002 14153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.574627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:45801 (local address 127.28.72.195:34135)
0923 17:26:27.575727 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:27.575785 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:27.577570 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:27.578343 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:27.578352 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:27.579181 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:27.579800 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:27.579818 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.580962 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.580967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:27.582322 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:27.582336 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.583077 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.583082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:27.583519 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.584190 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:26:27.584213 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:27.586779 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:27.590783 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:27.590801 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:27.590817 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:27.591357 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:27.591744 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:27.591750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:27.591755 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:27.591969 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:27.592516 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:27.592524 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:27.592889 (+ 365us) client_negotiation.cc:769] Sending connection context
0923 17:26:27.593169 (+ 280us) client_negotiation.cc:240] Negotiation successful
0923 17:26:27.593762 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":478,"threads_started":1}
I20260923 17:26:27.596544 14150 heartbeater.cc:347] Connected to a master server at 127.28.72.254:45801
I20260923 17:26:27.596880 14150 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:27.597682 14150 heartbeater.cc:511] Master 127.28.72.254:45801 requested a full tablet report, sending...
I20260923 17:26:27.599318 13693 ts_manager.cc:194] Registered new tserver with Master: 8c196ee2550a468ca08bd7d24100ac8b (127.28.72.195:36975)
I20260923 17:26:27.600248 13693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:34135
I20260923 17:26:27.602238 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:26:27.621374 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.607207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.28.72.254:45801)
0923 17:26:27.607401 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:27.607407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:27.607974 (+ 567us) server_negotiation.cc:407] Connection header received
0923 17:26:27.608178 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:27.608184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:27.608280 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:27.608392 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:27.609911 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.610681 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.612271 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.612781 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.612928 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.615832 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:27.615837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:27.615840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:27.615878 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:27.618495 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:27.619245 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:27.619251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:27.619254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:27.619388 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:27.619803 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:27.619810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:27.619817 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:27.620165 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:27.620426 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:27.620805 (+ 379us) server_negotiation.cc:299] Negotiation successful
0923 17:26:27.621047 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:26:27.626339 13692 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:26:27.633528 13692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:26:27.671075 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.647613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36975 (local address 127.0.0.1:60854)
0923 17:26:27.660672 (+ 13059us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:27.660725 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:27.661187 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:27.662384 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:27.662390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:27.662419 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:27.662894 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:27.662911 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.665086 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.665094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:27.667694 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:27.667709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.669685 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.669690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:27.669897 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.670273 (+ 376us) client_negotiation.cc:769] Sending connection context
0923 17:26:27.670431 (+ 158us) client_negotiation.cc:240] Negotiation successful
0923 17:26:27.670655 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12779,"thread_start_us":16154,"threads_started":1}
I20260923 17:26:27.671995 14163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.647672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60854 (local address 127.28.72.195:36975)
0923 17:26:27.648982 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:27.648992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:27.660886 (+ 11894us) server_negotiation.cc:407] Connection header received
0923 17:26:27.661521 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:27.661530 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:27.661816 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:27.661981 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:27.663145 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.664919 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.667909 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.669488 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.669659 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.669929 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:27.671397 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0923 17:26:27.671628 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1096,"thread_start_us":2051,"threads_started":1}
I20260923 17:26:27.672325 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.647628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:34033 (local address 127.0.0.1:36260)
0923 17:26:27.663437 (+ 15809us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:27.663459 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:27.663585 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:27.664379 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:27.664383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:27.664405 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:27.664733 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:27.664742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.667075 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.667080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:27.668885 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:27.668898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.671102 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.671134 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:27.671638 (+ 504us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.671675 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 17:26:27.671775 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 17:26:27.671931 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15671,"mutex_wait_us":256,"thread_start_us":15274,"threads_started":1}
I20260923 17:26:27.672370 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.647982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.28.72.193:34033)
0923 17:26:27.663407 (+ 15425us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:27.663418 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:27.663688 (+ 270us) server_negotiation.cc:407] Connection header received
0923 17:26:27.663812 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:27.663819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:27.664106 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:27.664330 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:27.664946 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.666906 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.669063 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.670511 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.670677 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.670916 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:27.671849 (+ 933us) server_negotiation.cc:299] Negotiation successful
0923 17:26:27.672046 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15201,"thread_start_us":15447,"threads_started":1}
I20260923 17:26:27.672595 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.649285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.28.72.194:38373)
0923 17:26:27.659724 (+ 10439us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:27.659740 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:27.664264 (+ 4524us) server_negotiation.cc:407] Connection header received
0923 17:26:27.664374 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:27.664381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:27.664609 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:27.664798 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:27.665453 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.666911 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.669480 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.671005 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.671172 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.671382 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:27.672048 (+ 666us) server_negotiation.cc:299] Negotiation successful
0923 17:26:27.672227 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10164,"thread_start_us":10464,"threads_started":1}
I20260923 17:26:27.673102 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.648747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:38373 (local address 127.0.0.1:35602)
0923 17:26:27.664114 (+ 15367us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:27.664132 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:27.664246 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:27.664798 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:27.664803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:27.664840 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:27.665281 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:27.665293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.667054 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.667060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:27.669285 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:27.669300 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.671526 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.671533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:27.671709 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.671751 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 17:26:27.671826 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 17:26:27.671944 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15229,"mutex_wait_us":28,"thread_start_us":11485,"threads_started":1}
I20260923 17:26:27.676501 14085 tablet_service.cc:1467] Processing CreateTablet for tablet 75aacb7b7bd44dad825f85a77341e54e (DEFAULT_TABLE table=test-table [id=7672a8394dc84fccaa09fb1644b4f34b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:27.676717 13817 tablet_service.cc:1467] Processing CreateTablet for tablet 75aacb7b7bd44dad825f85a77341e54e (DEFAULT_TABLE table=test-table [id=7672a8394dc84fccaa09fb1644b4f34b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:27.676717 13951 tablet_service.cc:1467] Processing CreateTablet for tablet 75aacb7b7bd44dad825f85a77341e54e (DEFAULT_TABLE table=test-table [id=7672a8394dc84fccaa09fb1644b4f34b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:27.678153 13817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75aacb7b7bd44dad825f85a77341e54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:27.678177 13951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75aacb7b7bd44dad825f85a77341e54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:27.678181 14085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75aacb7b7bd44dad825f85a77341e54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:27.692407 14169 tablet_bootstrap.cc:492] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b: Bootstrap starting.
I20260923 17:26:27.693642 14170 tablet_bootstrap.cc:492] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901: Bootstrap starting.
I20260923 17:26:27.694510 14171 tablet_bootstrap.cc:492] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d: Bootstrap starting.
I20260923 17:26:27.696556 14169 tablet_bootstrap.cc:654] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:27.698031 14169 log.cc:824] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:27.698985 14170 tablet_bootstrap.cc:654] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:27.699335 14171 tablet_bootstrap.cc:654] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:27.700301 14170 log.cc:824] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:27.700604 14171 log.cc:824] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:27.702677 14169 tablet_bootstrap.cc:492] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b: No bootstrap required, opened a new log
I20260923 17:26:27.703011 14169 ts_tablet_manager.cc:1397] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260923 17:26:27.703831 14170 tablet_bootstrap.cc:492] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901: No bootstrap required, opened a new log
I20260923 17:26:27.704101 14170 ts_tablet_manager.cc:1397] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 17:26:27.704751 14171 tablet_bootstrap.cc:492] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d: No bootstrap required, opened a new log
I20260923 17:26:27.705075 14171 ts_tablet_manager.cc:1397] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260923 17:26:27.713578 14171 raft_consensus.cc:348] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c196ee2550a468ca08bd7d24100ac8b" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36975 } } peers { permanent_uuid: "e8e7771d26124e639ac4e3f095a83a6d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34033 } } peers { permanent_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38373 } }
I20260923 17:26:27.714044 14171 raft_consensus.cc:374] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:27.714280 14171 raft_consensus.cc:729] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e7771d26124e639ac4e3f095a83a6d, State: Initialized, Role: FOLLOWER
I20260923 17:26:27.714526 14169 raft_consensus.cc:348] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c196ee2550a468ca08bd7d24100ac8b" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36975 } } peers { permanent_uuid: "e8e7771d26124e639ac4e3f095a83a6d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34033 } } peers { permanent_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38373 } }
I20260923 17:26:27.715004 14169 raft_consensus.cc:374] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:27.715147 14169 raft_consensus.cc:729] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c196ee2550a468ca08bd7d24100ac8b, State: Initialized, Role: FOLLOWER
I20260923 17:26:27.715054 14171 consensus_queue.cc:260] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c196ee2550a468ca08bd7d24100ac8b" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36975 } } peers { permanent_uuid: "e8e7771d26124e639ac4e3f095a83a6d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34033 } } peers { permanent_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38373 } }
I20260923 17:26:27.715296 14170 raft_consensus.cc:348] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c196ee2550a468ca08bd7d24100ac8b" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36975 } } peers { permanent_uuid: "e8e7771d26124e639ac4e3f095a83a6d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34033 } } peers { permanent_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38373 } }
I20260923 17:26:27.715803 14170 raft_consensus.cc:374] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:27.715951 14170 raft_consensus.cc:729] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fe1dd594a854b4ba87eb84f15ea7901, State: Initialized, Role: FOLLOWER
I20260923 17:26:27.716611 14170 consensus_queue.cc:260] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c196ee2550a468ca08bd7d24100ac8b" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36975 } } peers { permanent_uuid: "e8e7771d26124e639ac4e3f095a83a6d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34033 } } peers { permanent_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38373 } }
I20260923 17:26:27.717029 14171 ts_tablet_manager.cc:1428] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260923 17:26:27.715966 14169 consensus_queue.cc:260] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c196ee2550a468ca08bd7d24100ac8b" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36975 } } peers { permanent_uuid: "e8e7771d26124e639ac4e3f095a83a6d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34033 } } peers { permanent_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38373 } }
I20260923 17:26:27.717733 14016 heartbeater.cc:503] Master 127.28.72.254:45801 was elected leader, sending a full tablet report...
I20260923 17:26:27.724565 14169 ts_tablet_manager.cc:1428] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b: Time spent starting tablet: real 0.021s user 0.006s sys 0.012s
I20260923 17:26:27.727428 14150 heartbeater.cc:503] Master 127.28.72.254:45801 was elected leader, sending a full tablet report...
I20260923 17:26:27.728327 14170 ts_tablet_manager.cc:1428] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901: Time spent starting tablet: real 0.024s user 0.013s sys 0.000s
W20260923 17:26:27.769711 14017 tablet.cc:2369] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:26:27.822515 14151 tablet.cc:2369] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:26:27.845386 14176 raft_consensus.cc:482] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:26:27.845782 14176 raft_consensus.cc:504] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c196ee2550a468ca08bd7d24100ac8b" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36975 } } peers { permanent_uuid: "e8e7771d26124e639ac4e3f095a83a6d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34033 } } peers { permanent_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38373 } }
I20260923 17:26:27.847608 14176 leader_election.cc:288] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c196ee2550a468ca08bd7d24100ac8b (127.28.72.195:36975), e8e7771d26124e639ac4e3f095a83a6d (127.28.72.193:34033)
I20260923 17:26:27.855389 14163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.848185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:38399 (local address 127.28.72.195:36975)
0923 17:26:27.848415 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:27.848422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:27.849204 (+ 782us) server_negotiation.cc:407] Connection header received
0923 17:26:27.849377 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:27.849382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:27.849470 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:27.849567 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:27.850323 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.851064 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.853525 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.854390 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.854604 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.854679 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:27.854975 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 17:26:27.855129 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:26:27.855443 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.848085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:52443 (local address 127.28.72.193:34033)
0923 17:26:27.848417 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:27.848426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:27.849367 (+ 941us) server_negotiation.cc:407] Connection header received
0923 17:26:27.849565 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:27.849568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:27.849630 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:27.849724 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:27.850479 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.851385 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.853603 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.854405 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.854514 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.854568 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:27.855019 (+ 451us) server_negotiation.cc:299] Negotiation successful
0923 17:26:27.855187 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:26:27.855600 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.848066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36975 (local address 127.28.72.194:38399)
0923 17:26:27.849067 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:27.849105 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:27.849236 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:27.849597 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:27.849601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:27.849634 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:27.850156 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:27.850170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.851245 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.851254 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:27.853366 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:27.853383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.854629 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.854636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:27.854800 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.854828 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 17:26:27.854946 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 17:26:27.855087 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":57,"spinlock_wait_cycles":31488,"thread_start_us":641,"threads_started":1}
I20260923 17:26:27.855629 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.848180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:34033 (local address 127.28.72.194:52443)
0923 17:26:27.849205 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:27.849224 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:27.849391 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:27.849762 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:27.849767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:27.849804 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:27.850322 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:27.850334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.851530 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.851539 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:27.853408 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:27.853422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.854728 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.854732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:27.854863 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.854884 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:26:27.854988 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 17:26:27.855113 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":5,"thread_start_us":478,"threads_started":1}
I20260923 17:26:27.856809 13837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75aacb7b7bd44dad825f85a77341e54e" candidate_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e7771d26124e639ac4e3f095a83a6d" is_pre_election: true
I20260923 17:26:27.856882 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75aacb7b7bd44dad825f85a77341e54e" candidate_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c196ee2550a468ca08bd7d24100ac8b" is_pre_election: true
I20260923 17:26:27.857376 14105 raft_consensus.cc:2454] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fe1dd594a854b4ba87eb84f15ea7901 in term 0.
I20260923 17:26:27.857362 13837 raft_consensus.cc:2454] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fe1dd594a854b4ba87eb84f15ea7901 in term 0.
I20260923 17:26:27.858085 13905 leader_election.cc:302] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8c196ee2550a468ca08bd7d24100ac8b, 8fe1dd594a854b4ba87eb84f15ea7901, e8e7771d26124e639ac4e3f095a83a6d; no voters:
I20260923 17:26:27.858516 14176 raft_consensus.cc:2781] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:26:27.858667 14176 raft_consensus.cc:482] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:26:27.858795 14176 raft_consensus.cc:3037] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:27.861132 14176 raft_consensus.cc:504] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c196ee2550a468ca08bd7d24100ac8b" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36975 } } peers { permanent_uuid: "e8e7771d26124e639ac4e3f095a83a6d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34033 } } peers { permanent_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38373 } }
I20260923 17:26:27.861788 14176 leader_election.cc:288] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [CANDIDATE]: Term 1 election: Requested vote from peers 8c196ee2550a468ca08bd7d24100ac8b (127.28.72.195:36975), e8e7771d26124e639ac4e3f095a83a6d (127.28.72.193:34033)
I20260923 17:26:27.862394 13837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75aacb7b7bd44dad825f85a77341e54e" candidate_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e7771d26124e639ac4e3f095a83a6d"
I20260923 17:26:27.862442 14105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75aacb7b7bd44dad825f85a77341e54e" candidate_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c196ee2550a468ca08bd7d24100ac8b"
I20260923 17:26:27.862629 13837 raft_consensus.cc:3037] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:27.862811 14105 raft_consensus.cc:3037] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:27.865072 13837 raft_consensus.cc:2454] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fe1dd594a854b4ba87eb84f15ea7901 in term 1.
I20260923 17:26:27.865157 14105 raft_consensus.cc:2454] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fe1dd594a854b4ba87eb84f15ea7901 in term 1.
I20260923 17:26:27.865676 13905 leader_election.cc:302] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fe1dd594a854b4ba87eb84f15ea7901, e8e7771d26124e639ac4e3f095a83a6d; no voters:
I20260923 17:26:27.866024 14176 raft_consensus.cc:2781] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:27.866479 14176 raft_consensus.cc:686] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [term 1 LEADER]: Becoming Leader. State: Replica: 8fe1dd594a854b4ba87eb84f15ea7901, State: Running, Role: LEADER
I20260923 17:26:27.866997 14176 consensus_queue.cc:237] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c196ee2550a468ca08bd7d24100ac8b" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36975 } } peers { permanent_uuid: "e8e7771d26124e639ac4e3f095a83a6d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34033 } } peers { permanent_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38373 } }
I20260923 17:26:27.871881 13693 catalog_manager.cc:5495] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fe1dd594a854b4ba87eb84f15ea7901 (127.28.72.194). New cstate: current_term: 1 leader_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c196ee2550a468ca08bd7d24100ac8b" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8e7771d26124e639ac4e3f095a83a6d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fe1dd594a854b4ba87eb84f15ea7901" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 38373 } health_report { overall_health: HEALTHY } } }
I20260923 17:26:27.912417 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.906586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.28.72.194:38373)
0923 17:26:27.906766 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:27.906773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:27.906839 (+ 66us) server_negotiation.cc:407] Connection header received
0923 17:26:27.907186 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:27.907192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:27.907288 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:27.907513 (+ 225us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:26:27.907887 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.908680 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.910059 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.910479 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.910619 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.911250 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:26:27.911788 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:26:27.911952 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:27.912054 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 17:26:27.912142 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 17:26:27.921063 13837 raft_consensus.cc:1261] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d [term 1 FOLLOWER]: Refusing update from remote peer 8fe1dd594a854b4ba87eb84f15ea7901: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:26:27.921063 14105 raft_consensus.cc:1261] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b [term 1 FOLLOWER]: Refusing update from remote peer 8fe1dd594a854b4ba87eb84f15ea7901: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:26:27.922092 14180 consensus_queue.cc:1035] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8e7771d26124e639ac4e3f095a83a6d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:26:27.922477 14176 consensus_queue.cc:1035] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c196ee2550a468ca08bd7d24100ac8b" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:26:27.934666 14183 mvcc.cc:205] Tried to move back new op lower bound from 7332595252913741824 to 7332595252707536896. Current Snapshot: MvccSnapshot[applied={T|T < 7332595252913741824}]
I20260923 17:26:27.949658 13693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:26:27.950024 13693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:26:27.954149 13693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 548461735def4164969d00e31f7cfc3c: Sending DeleteTablet for 3 replicas of tablet 75aacb7b7bd44dad825f85a77341e54e
I20260923 17:26:27.955559 13817 tablet_service.cc:1514] Processing DeleteTablet for tablet 75aacb7b7bd44dad825f85a77341e54e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:27 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:36260
I20260923 17:26:27.955559 13951 tablet_service.cc:1514] Processing DeleteTablet for tablet 75aacb7b7bd44dad825f85a77341e54e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:27 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:35602
I20260923 17:26:27.955559 14085 tablet_service.cc:1514] Processing DeleteTablet for tablet 75aacb7b7bd44dad825f85a77341e54e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:27 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:60854
I20260923 17:26:27.956652 14195 tablet_replica.cc:330] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901: stopping tablet replica
I20260923 17:26:27.956846 14196 tablet_replica.cc:330] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d: stopping tablet replica
I20260923 17:26:27.957261 14194 tablet_replica.cc:330] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b: stopping tablet replica
I20260923 17:26:27.957476 14196 raft_consensus.cc:2229] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:26:27.957688 14194 raft_consensus.cc:2229] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:26:27.957846 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 13754
I20260923 17:26:27.958148 14196 raft_consensus.cc:2258] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:27.958230 14194 raft_consensus.cc:2258] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:27.958870 14195 raft_consensus.cc:2229] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:26:27.959578 14195 raft_consensus.cc:2258] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:27.960580 14196 ts_tablet_manager.cc:1905] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:27.962695 14195 ts_tablet_manager.cc:1905] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:27.966454 14194 ts_tablet_manager.cc:1905] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:27.969125 14196 ts_tablet_manager.cc:1918] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:27.969383 14196 log.cc:1196] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/75aacb7b7bd44dad825f85a77341e54e
I20260923 17:26:27.970050 14196 ts_tablet_manager.cc:1939] T 75aacb7b7bd44dad825f85a77341e54e P e8e7771d26124e639ac4e3f095a83a6d: Deleting consensus metadata
I20260923 17:26:27.970402 14195 ts_tablet_manager.cc:1918] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:27.970623 14195 log.cc:1196] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/75aacb7b7bd44dad825f85a77341e54e
I20260923 17:26:27.971299 14195 ts_tablet_manager.cc:1939] T 75aacb7b7bd44dad825f85a77341e54e P 8fe1dd594a854b4ba87eb84f15ea7901: Deleting consensus metadata
I20260923 17:26:27.972729 13678 catalog_manager.cc:4841] TS e8e7771d26124e639ac4e3f095a83a6d (127.28.72.193:34033): tablet 75aacb7b7bd44dad825f85a77341e54e (table test-table [id=7672a8394dc84fccaa09fb1644b4f34b]) successfully deleted
I20260923 17:26:27.973104 13677 catalog_manager.cc:4841] TS 8fe1dd594a854b4ba87eb84f15ea7901 (127.28.72.194:38373): tablet 75aacb7b7bd44dad825f85a77341e54e (table test-table [id=7672a8394dc84fccaa09fb1644b4f34b]) successfully deleted
I20260923 17:26:27.977793 14194 ts_tablet_manager.cc:1918] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:27.978013 14194 log.cc:1196] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/75aacb7b7bd44dad825f85a77341e54e
I20260923 17:26:27.978627 14194 ts_tablet_manager.cc:1939] T 75aacb7b7bd44dad825f85a77341e54e P 8c196ee2550a468ca08bd7d24100ac8b: Deleting consensus metadata
I20260923 17:26:27.980707 13680 catalog_manager.cc:4841] TS 8c196ee2550a468ca08bd7d24100ac8b (127.28.72.195:36975): tablet 75aacb7b7bd44dad825f85a77341e54e (table test-table [id=7672a8394dc84fccaa09fb1644b4f34b]) successfully deleted
Sep 23 17:26:27 dist-test-slave-j0m4 krb5kdc[13648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:26:27.993227 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:27.959498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36270 (local address 127.28.72.193:34033)
0923 17:26:27.959696 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:27.959702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:27.974612 (+ 14910us) server_negotiation.cc:407] Connection header received
0923 17:26:27.974870 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:27.974876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:27.974964 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:27.975077 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:27.976783 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.977943 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.979926 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:27.980577 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:27.980705 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:27.986074 (+ 5369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:27.986094 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:27.986116 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:27.986175 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:27.990222 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:27.990994 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:27.991005 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:27.991008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:27.991204 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:27.991569 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:27.991574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:27.991578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:27.992066 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:27.992372 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:27.992596 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 17:26:27.992837 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:26:27.993968 13877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:28.087424 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 13754
I20260923 17:26:28.106293 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 13888
Sep 23 17:26:28 dist-test-slave-j0m4 krb5kdc[13648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:26:28.125702 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:28.107216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.28.72.194:38373)
0923 17:26:28.107432 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:28.107436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:28.107559 (+ 123us) server_negotiation.cc:407] Connection header received
0923 17:26:28.107781 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:28.107785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:28.107854 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:28.107939 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:28.109707 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:28.110575 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:28.112652 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:28.113188 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:28.113348 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:28.118874 (+ 5526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:28.118893 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:28.118910 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:28.118958 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:28.122365 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:28.123378 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:28.123387 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:28.123390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:28.123541 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:28.124005 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:28.124018 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:28.124020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:28.124392 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:28.124778 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:28.125050 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 17:26:28.125280 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:26:28.126459 14011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:28.242707 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 13888
I20260923 17:26:28.264510 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 14022
Sep 23 17:26:28 dist-test-slave-j0m4 krb5kdc[13648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:26:28.282277 14163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:28.265307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.28.72.195:36975)
0923 17:26:28.265533 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:28.265537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:28.265555 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:26:28.265719 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:28.265723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:28.265784 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:28.265870 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:28.267648 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:28.268516 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:28.269996 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:28.270480 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:28.270591 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:28.275840 (+ 5249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:28.275890 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:28.275908 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:28.275953 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:28.279233 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:28.280075 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:28.280083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:28.280086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:28.280240 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:28.280624 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:28.280629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:28.280632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:28.280992 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:28.281305 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:28.281688 (+ 383us) server_negotiation.cc:299] Negotiation successful
0923 17:26:28.281920 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:26:28.282907 14145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:28.374797 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 14022
I20260923 17:26:28.393635 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 13661
I20260923 17:26:28.394812 13723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:28.507493 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 13661
2026-09-23T17:26:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4615 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T17:26:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:26:28Z Disabled control of system clock
I20260923 17:26:28.588912 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:42277
--webserver_interface=127.28.72.254
--webserver_port=0
--builtin_ntp_servers=127.28.72.212:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:42277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:26:28.937980 14205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:28.938310 14205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:28.938465 14205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:28.945959 14205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:26:28.946090 14205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:28.946168 14205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:28.946257 14205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:26:28.946324 14205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:26:28.956014 14205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:42277
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:42277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:28.957432 14205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:28.959410 14205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:28.972580 14210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:28.972491 14213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:28.972589 14211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:28.973696 14205 server_base.cc:1034] running on GCE node
I20260923 17:26:28.974789 14205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:28.976673 14205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:28.978003 14205 hybrid_clock.cc:648] HybridClock initialized: now 1790184388977954 us; error 42 us; skew 500 ppm
I20260923 17:26:28.978457 14205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:26:28.981350 14205 webserver.cc:460] Webserver started at http://127.28.72.254:40985/ using document root <none> and password file <none>
I20260923 17:26:28.982312 14205 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:28.982461 14205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:28.982822 14205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:28.986249 14205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "9a936bc3c5f548a3a105bf45ae77e60b"
format_stamp: "Formatted at 2026-09-23 17:26:28 on dist-test-slave-j0m4"
server_key: "6b7d7bd9465a19449aec6dec72f11b71"
server_key_iv: "1e37922b80cde948c3c9199d7e85ba39"
server_key_version: "encryptionkey@0"
I20260923 17:26:28.987264 14205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "9a936bc3c5f548a3a105bf45ae77e60b"
format_stamp: "Formatted at 2026-09-23 17:26:28 on dist-test-slave-j0m4"
server_key: "6b7d7bd9465a19449aec6dec72f11b71"
server_key_iv: "1e37922b80cde948c3c9199d7e85ba39"
server_key_version: "encryptionkey@0"
I20260923 17:26:28.993134 14205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:26:28.997320 14219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:28.998914 14205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:26:28.999218 14205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "9a936bc3c5f548a3a105bf45ae77e60b"
format_stamp: "Formatted at 2026-09-23 17:26:28 on dist-test-slave-j0m4"
server_key: "6b7d7bd9465a19449aec6dec72f11b71"
server_key_iv: "1e37922b80cde948c3c9199d7e85ba39"
server_key_version: "encryptionkey@0"
I20260923 17:26:28.999552 14205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:29.017457 14205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:29.018476 14205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:29.018824 14205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:29.038997 14205 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:42277
I20260923 17:26:29.039005 14270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:42277 every 8 connection(s)
I20260923 17:26:29.040534 14205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:26:29.044814 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 14205
I20260923 17:26:29.045046 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:26:29.045262 14271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:29.045641 28963 external_mini_cluster.cc:1428] Setting key 415751f36c70336eb0c647c658db315b
I20260923 17:26:29.059557 14271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b: Bootstrap starting.
I20260923 17:26:29.061808 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:29.048519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.28.72.254:42277)
0923 17:26:29.049903 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:29.049916 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:29.049969 (+ 53us) server_negotiation.cc:407] Connection header received
0923 17:26:29.052380 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:29.052416 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:29.053022 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:29.053501 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:29.054116 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:29.055545 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:29.057060 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:29.057809 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:29.058025 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:29.059328 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:29.059336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:29.059353 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:29.059999 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:29.060092 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:29.060120 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:29.060282 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:29.060472 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 17:26:29.060810 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":481,"threads_started":1}
I20260923 17:26:29.063175 14271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:29.064565 14271 log.cc:824] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:29.068084 14271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b: No bootstrap required, opened a new log
I20260923 17:26:29.076817 14271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a936bc3c5f548a3a105bf45ae77e60b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42277 } }
I20260923 17:26:29.077240 14271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:29.077343 14271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a936bc3c5f548a3a105bf45ae77e60b, State: Initialized, Role: FOLLOWER
I20260923 17:26:29.078078 14271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a936bc3c5f548a3a105bf45ae77e60b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42277 } }
I20260923 17:26:29.078338 14271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:26:29.078495 14271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:26:29.078636 14271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:29.081503 14271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a936bc3c5f548a3a105bf45ae77e60b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42277 } }
I20260923 17:26:29.082121 14271 leader_election.cc:302] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a936bc3c5f548a3a105bf45ae77e60b; no voters:
I20260923 17:26:29.082747 14271 leader_election.cc:288] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:26:29.082953 14276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:29.083732 14276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [term 1 LEADER]: Becoming Leader. State: Replica: 9a936bc3c5f548a3a105bf45ae77e60b, State: Running, Role: LEADER
I20260923 17:26:29.084370 14276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a936bc3c5f548a3a105bf45ae77e60b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42277 } }
I20260923 17:26:29.085067 14271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:26:29.087577 14278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a936bc3c5f548a3a105bf45ae77e60b. Latest consensus state: current_term: 1 leader_uuid: "9a936bc3c5f548a3a105bf45ae77e60b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a936bc3c5f548a3a105bf45ae77e60b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42277 } } }
I20260923 17:26:29.087669 14277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a936bc3c5f548a3a105bf45ae77e60b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a936bc3c5f548a3a105bf45ae77e60b" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 42277 } } }
I20260923 17:26:29.088300 14277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:29.088297 14278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:29.092280 14284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:26:29.097833 14284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:26:29.108052 14284 catalog_manager.cc:1347] Generated new cluster ID: a4f31420796c4ebd9db9a1e3dc93cff5
I20260923 17:26:29.108197 14284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:26:29.132927 14284 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:26:29.134927 14284 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:26:29.144101 14284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b: Generated new TSK 0
I20260923 17:26:29.145296 14284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:26:29.154754 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42277
--builtin_ntp_servers=127.28.72.212:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:26:29.509397 14295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:29.509747 14295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:29.509903 14295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:29.517544 14295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:29.517724 14295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:29.517906 14295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:26:29.527962 14295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:29.529422 14295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:29.531812 14295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:29.546974 14303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:29.547099 14300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:29.547079 14301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:29.549059 14295 server_base.cc:1034] running on GCE node
I20260923 17:26:29.549810 14295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:29.551007 14295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:29.552246 14295 hybrid_clock.cc:648] HybridClock initialized: now 1790184389552194 us; error 52 us; skew 500 ppm
I20260923 17:26:29.552687 14295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:26:29.555565 14295 webserver.cc:460] Webserver started at http://127.28.72.193:35481/ using document root <none> and password file <none>
I20260923 17:26:29.556356 14295 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:29.556495 14295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:29.556851 14295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:29.560111 14295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "680b92d061514308991709809730056d"
format_stamp: "Formatted at 2026-09-23 17:26:29 on dist-test-slave-j0m4"
server_key: "4605b122a9788a50e34421d0a0ef0dc1"
server_key_iv: "9a683af96f94526e43c1cd3a067134d7"
server_key_version: "encryptionkey@0"
I20260923 17:26:29.561008 14295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "680b92d061514308991709809730056d"
format_stamp: "Formatted at 2026-09-23 17:26:29 on dist-test-slave-j0m4"
server_key: "4605b122a9788a50e34421d0a0ef0dc1"
server_key_iv: "9a683af96f94526e43c1cd3a067134d7"
server_key_version: "encryptionkey@0"
I20260923 17:26:29.566847 14295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:26:29.570909 14309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:29.572727 14295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:26:29.572996 14295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "680b92d061514308991709809730056d"
format_stamp: "Formatted at 2026-09-23 17:26:29 on dist-test-slave-j0m4"
server_key: "4605b122a9788a50e34421d0a0ef0dc1"
server_key_iv: "9a683af96f94526e43c1cd3a067134d7"
server_key_version: "encryptionkey@0"
I20260923 17:26:29.573267 14295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:29.588706 14295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:29.589695 14295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:29.590034 14295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:29.591076 14295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:29.592954 14295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:29.593081 14295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:29.593223 14295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:29.593310 14295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:29.629748 14295 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:38227
I20260923 17:26:29.629869 14421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:38227 every 8 connection(s)
I20260923 17:26:29.631217 14295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:26:29.631563 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 14295
I20260923 17:26:29.631780 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:26:29.632452 28963 external_mini_cluster.cc:1428] Setting key 6c2f9b088352a07ac96e0bfa8ac527eb
I20260923 17:26:29.636053 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42277
--builtin_ntp_servers=127.28.72.212:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:26:29.649298 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:29.634297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:40529 (local address 127.28.72.254:42277)
0923 17:26:29.634795 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:29.634802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:29.636455 (+ 1653us) server_negotiation.cc:407] Connection header received
0923 17:26:29.639973 (+ 3518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:29.639978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:29.640075 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:29.640187 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:29.641907 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:29.642816 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:29.646023 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:29.646453 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:29.646608 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:29.648366 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:29.648372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:29.648375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:29.648397 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:29.648436 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:29.648448 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:29.648562 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:29.648810 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 17:26:29.648932 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":258,"threads_started":1}
I20260923 17:26:29.650434 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:29.635031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:42277 (local address 127.28.72.193:40529)
0923 17:26:29.636249 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:29.636312 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:29.639544 (+ 3232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:29.640532 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:29.640548 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:29.640687 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:29.641675 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:29.641707 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:29.643046 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:29.643054 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:29.645810 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:29.645829 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:29.646719 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:29.646726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:29.647177 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:29.647982 (+ 805us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:26:29.647985 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:29.648030 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:26:29.648121 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:29.648558 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:29.648571 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:29.648581 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 17:26:29.648786 (+ 205us) client_negotiation.cc:240] Negotiation successful
0923 17:26:29.649241 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":308,"threads_started":1}
I20260923 17:26:29.652360 14422 heartbeater.cc:347] Connected to a master server at 127.28.72.254:42277
I20260923 17:26:29.652721 14422 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:29.653612 14422 heartbeater.cc:511] Master 127.28.72.254:42277 requested a full tablet report, sending...
I20260923 17:26:29.656222 14236 ts_manager.cc:194] Registered new tserver with Master: 680b92d061514308991709809730056d (127.28.72.193:38227)
I20260923 17:26:29.658517 14236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.72.193:40529
W20260923 17:26:29.987469 14426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:29.987759 14426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:29.987856 14426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:29.995405 14426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:29.995522 14426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:29.995715 14426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:26:30.005344 14426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:30.006739 14426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:30.008651 14426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:30.022737 14431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:30.022732 14432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:30.022959 14434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:30.023584 14426 server_base.cc:1034] running on GCE node
I20260923 17:26:30.024358 14426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:30.025437 14426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:30.026706 14426 hybrid_clock.cc:648] HybridClock initialized: now 1790184390026650 us; error 45 us; skew 500 ppm
I20260923 17:26:30.027103 14426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:26:30.029631 14426 webserver.cc:460] Webserver started at http://127.28.72.194:39159/ using document root <none> and password file <none>
I20260923 17:26:30.030366 14426 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:30.030514 14426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:30.030840 14426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:30.033912 14426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "7fe936d76aed4bb1b82aee1f91c48336"
format_stamp: "Formatted at 2026-09-23 17:26:30 on dist-test-slave-j0m4"
server_key: "cec59f9ddf1c62f2405f799c815ff455"
server_key_iv: "b3ca1c56896d01be436cb6e3c54f21d8"
server_key_version: "encryptionkey@0"
I20260923 17:26:30.034773 14426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "7fe936d76aed4bb1b82aee1f91c48336"
format_stamp: "Formatted at 2026-09-23 17:26:30 on dist-test-slave-j0m4"
server_key: "cec59f9ddf1c62f2405f799c815ff455"
server_key_iv: "b3ca1c56896d01be436cb6e3c54f21d8"
server_key_version: "encryptionkey@0"
I20260923 17:26:30.040280 14426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:26:30.044104 14440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:30.045730 14426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:26:30.045921 14426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "7fe936d76aed4bb1b82aee1f91c48336"
format_stamp: "Formatted at 2026-09-23 17:26:30 on dist-test-slave-j0m4"
server_key: "cec59f9ddf1c62f2405f799c815ff455"
server_key_iv: "b3ca1c56896d01be436cb6e3c54f21d8"
server_key_version: "encryptionkey@0"
I20260923 17:26:30.046161 14426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:30.066929 14426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:30.067895 14426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:30.068238 14426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:30.069306 14426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:30.070859 14426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:30.071000 14426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:30.071178 14426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:30.071282 14426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:30.107323 14426 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:44325
I20260923 17:26:30.107364 14552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:44325 every 8 connection(s)
I20260923 17:26:30.108723 14426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:26:30.118412 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 14426
I20260923 17:26:30.118709 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:26:30.119329 28963 external_mini_cluster.cc:1428] Setting key e4efb5b7f53648d86a7553b6ab75de7f
I20260923 17:26:30.122696 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.111545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:48819 (local address 127.28.72.254:42277)
0923 17:26:30.111772 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:30.111779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:30.113276 (+ 1497us) server_negotiation.cc:407] Connection header received
0923 17:26:30.115102 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:30.115107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:30.115250 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:30.115373 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:30.116502 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.117310 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.119156 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.119801 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.119957 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.121725 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:30.121731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:30.121735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:30.121759 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:30.121802 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:30.121819 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:30.121918 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:30.122162 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 17:26:30.122268 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:26:30.122958 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42277
--builtin_ntp_servers=127.28.72.212:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:26:30.123733 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.112124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:42277 (local address 127.28.72.194:48819)
0923 17:26:30.113100 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:30.113147 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:30.114793 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:30.115600 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:30.115609 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:30.115706 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:30.116309 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:30.116327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.117464 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.117468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:30.118941 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:30.118978 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.120103 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.120111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:30.120400 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.121254 (+ 854us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:26:30.121258 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:30.121320 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:26:30.121459 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:30.121926 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:30.121935 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:30.121942 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:26:30.122109 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 17:26:30.122447 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":299,"threads_started":1}
I20260923 17:26:30.125322 14553 heartbeater.cc:347] Connected to a master server at 127.28.72.254:42277
I20260923 17:26:30.125679 14553 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:30.126680 14553 heartbeater.cc:511] Master 127.28.72.254:42277 requested a full tablet report, sending...
I20260923 17:26:30.128584 14236 ts_manager.cc:194] Registered new tserver with Master: 7fe936d76aed4bb1b82aee1f91c48336 (127.28.72.194:44325)
I20260923 17:26:30.129724 14236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.72.194:48819
W20260923 17:26:30.465524 14556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:30.465864 14556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:30.465956 14556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:30.473665 14556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:30.473799 14556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:30.473912 14556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:26:30.483762 14556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:30.485074 14556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:30.487066 14556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:30.501960 14562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:30.501955 14561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:30.502130 14564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:30.503232 14556 server_base.cc:1034] running on GCE node
I20260923 17:26:30.504086 14556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:30.505208 14556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:30.506449 14556 hybrid_clock.cc:648] HybridClock initialized: now 1790184390506402 us; error 35 us; skew 500 ppm
I20260923 17:26:30.506863 14556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:26:30.509526 14556 webserver.cc:460] Webserver started at http://127.28.72.195:42147/ using document root <none> and password file <none>
I20260923 17:26:30.510327 14556 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:30.510464 14556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:30.510854 14556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:30.514032 14556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "5b060851eac84964bf69242d77574667"
format_stamp: "Formatted at 2026-09-23 17:26:30 on dist-test-slave-j0m4"
server_key: "0e27d01e55742c1d1fc8a56ffeebd516"
server_key_iv: "8c486a6cd1f472dd45ff231fb5acb917"
server_key_version: "encryptionkey@0"
I20260923 17:26:30.514919 14556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "5b060851eac84964bf69242d77574667"
format_stamp: "Formatted at 2026-09-23 17:26:30 on dist-test-slave-j0m4"
server_key: "0e27d01e55742c1d1fc8a56ffeebd516"
server_key_iv: "8c486a6cd1f472dd45ff231fb5acb917"
server_key_version: "encryptionkey@0"
I20260923 17:26:30.520780 14556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:26:30.525277 14570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:30.526934 14556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260923 17:26:30.527189 14556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "5b060851eac84964bf69242d77574667"
format_stamp: "Formatted at 2026-09-23 17:26:30 on dist-test-slave-j0m4"
server_key: "0e27d01e55742c1d1fc8a56ffeebd516"
server_key_iv: "8c486a6cd1f472dd45ff231fb5acb917"
server_key_version: "encryptionkey@0"
I20260923 17:26:30.527464 14556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:30.560561 14556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:30.561558 14556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:30.561902 14556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:30.563069 14556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:30.564896 14556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:30.564998 14556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:30.565176 14556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:30.565256 14556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:30.603407 14556 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:36833
I20260923 17:26:30.603507 14682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:36833 every 8 connection(s)
I20260923 17:26:30.604950 14556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:26:30.611421 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 14556
I20260923 17:26:30.611639 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:26:30.612164 28963 external_mini_cluster.cc:1428] Setting key 240dfa347f5e063735e28f45d4c1ff3c
I20260923 17:26:30.618942 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.607712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:48091 (local address 127.28.72.254:42277)
0923 17:26:30.607892 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:30.607896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:30.609362 (+ 1466us) server_negotiation.cc:407] Connection header received
0923 17:26:30.611474 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:30.611481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:30.611595 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:30.611733 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:30.613080 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.614254 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.616270 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.616690 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.616841 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.618176 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:30.618182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:30.618186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:30.618213 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:30.618249 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:30.618264 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:30.618351 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:30.618575 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 17:26:30.618658 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:26:30.620054 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.608263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:42277 (local address 127.28.72.195:48091)
0923 17:26:30.609214 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:30.609251 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:30.611079 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:30.611982 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:30.611991 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:30.612088 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:30.612862 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:30.612896 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.614431 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.614436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:30.616104 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:30.616117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.616928 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.616932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:30.617150 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.617829 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:26:30.617832 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:30.617879 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:26:30.617975 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:30.618361 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:30.618369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:30.618375 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:26:30.618545 (+ 170us) client_negotiation.cc:240] Negotiation successful
0923 17:26:30.618863 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":287,"threads_started":1}
I20260923 17:26:30.621608 14683 heartbeater.cc:347] Connected to a master server at 127.28.72.254:42277
I20260923 17:26:30.621924 14683 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:30.622748 14683 heartbeater.cc:511] Master 127.28.72.254:42277 requested a full tablet report, sending...
I20260923 17:26:30.624450 14236 ts_manager.cc:194] Registered new tserver with Master: 5b060851eac84964bf69242d77574667 (127.28.72.195:36833)
I20260923 17:26:30.625443 14236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.72.195:48091
I20260923 17:26:30.628286 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:26:30.639974 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.633077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56430 (local address 127.28.72.254:42277)
0923 17:26:30.633293 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:30.633298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:30.633894 (+ 596us) server_negotiation.cc:407] Connection header received
0923 17:26:30.634122 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:30.634128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:30.634235 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:30.634382 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:30.634906 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.635722 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.637267 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.637728 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.637893 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.639171 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:30.639177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:30.639182 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:30.639212 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:30.639249 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:30.639263 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:30.639374 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:30.639525 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 17:26:30.639651 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 17:26:30.644996 14236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:26:30.652477 14236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:26:30.662336 14422 heartbeater.cc:503] Master 127.28.72.254:42277 was elected leader, sending a full tablet report...
I20260923 17:26:30.691257 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.667937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:44325 (local address 127.0.0.1:46224)
0923 17:26:30.678851 (+ 10914us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:30.678938 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:30.679251 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:30.682156 (+ 2905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:30.682165 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:30.682208 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:30.682868 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:30.682886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.685552 (+ 2666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.685560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:30.688049 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:30.688065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.690030 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.690036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:30.690313 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.690381 (+ 68us) client_negotiation.cc:769] Sending connection context
0923 17:26:30.690556 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 17:26:30.690786 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10012,"mutex_wait_us":94,"thread_start_us":17128,"threads_started":1}
I20260923 17:26:30.691226 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.668225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.28.72.194:44325)
0923 17:26:30.681177 (+ 12952us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:30.681192 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:30.681239 (+ 47us) server_negotiation.cc:407] Connection header received
0923 17:26:30.681394 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:30.681406 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:30.681810 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:30.682132 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:30.683409 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.685397 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.688369 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.689793 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.689966 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.690304 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:30.690615 (+ 311us) server_negotiation.cc:299] Negotiation successful
0923 17:26:30.690797 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12674,"thread_start_us":12729,"threads_started":1}
I20260923 17:26:30.691831 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.668980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.28.72.193:38227)
0923 17:26:30.681730 (+ 12750us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:30.681737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:30.681764 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:26:30.681846 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:30.681851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:30.682088 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:30.682315 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:30.683577 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.685129 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.688248 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.689911 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.690075 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.690404 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:30.690936 (+ 532us) server_negotiation.cc:299] Negotiation successful
0923 17:26:30.691163 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12571,"thread_start_us":12426,"threads_started":1}
I20260923 17:26:30.692162 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.668323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38227 (local address 127.0.0.1:39164)
0923 17:26:30.679684 (+ 11361us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:30.679708 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:30.679829 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:30.682596 (+ 2767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:30.682604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:30.682639 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:30.683342 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:30.683355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.685368 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.685378 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:30.687974 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:30.687995 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.690540 (+ 2545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.690547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:30.690719 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.690753 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:26:30.691622 (+ 869us) client_negotiation.cc:240] Negotiation successful
0923 17:26:30.691807 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11196,"mutex_wait_us":244,"thread_start_us":9426,"threads_started":1}
I20260923 17:26:30.696472 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.668784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36833 (local address 127.0.0.1:48814)
0923 17:26:30.685392 (+ 16608us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:30.685421 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:30.685647 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:30.686771 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:30.686776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:30.686815 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:30.687278 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:30.687292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.689414 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.689419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:30.691626 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:30.691645 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.695590 (+ 3945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.695600 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:30.695814 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.695854 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 17:26:30.695958 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 17:26:30.696124 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16460,"thread_start_us":10565,"threads_started":1}
I20260923 17:26:30.696653 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.669480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.28.72.195:36833)
0923 17:26:30.684936 (+ 15456us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:30.684946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:30.685527 (+ 581us) server_negotiation.cc:407] Connection header received
0923 17:26:30.685869 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:30.685882 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:30.686225 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:30.686460 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:30.687493 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.689221 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.691904 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.695350 (+ 3446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.695523 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.695802 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:30.696058 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 17:26:30.696240 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13706,"thread_start_us":13523,"threads_started":1}
I20260923 17:26:30.696293 14488 tablet_service.cc:1467] Processing CreateTablet for tablet acf8c3d34f5a462196e045732b804693 (DEFAULT_TABLE table=test-table [id=f7c72fa250944d94a50dd5d3069b927a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:30.696707 14357 tablet_service.cc:1467] Processing CreateTablet for tablet acf8c3d34f5a462196e045732b804693 (DEFAULT_TABLE table=test-table [id=f7c72fa250944d94a50dd5d3069b927a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:30.697911 14488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf8c3d34f5a462196e045732b804693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:30.698365 14357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf8c3d34f5a462196e045732b804693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:30.701246 14618 tablet_service.cc:1467] Processing CreateTablet for tablet acf8c3d34f5a462196e045732b804693 (DEFAULT_TABLE table=test-table [id=f7c72fa250944d94a50dd5d3069b927a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:30.702893 14618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf8c3d34f5a462196e045732b804693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:30.711395 14701 tablet_bootstrap.cc:492] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d: Bootstrap starting.
I20260923 17:26:30.714774 14701 tablet_bootstrap.cc:654] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:30.715934 14702 tablet_bootstrap.cc:492] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336: Bootstrap starting.
I20260923 17:26:30.716221 14701 log.cc:824] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:30.719362 14704 tablet_bootstrap.cc:492] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667: Bootstrap starting.
I20260923 17:26:30.720937 14702 tablet_bootstrap.cc:654] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:30.722373 14702 log.cc:824] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:30.722921 14704 tablet_bootstrap.cc:654] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:30.723505 14701 tablet_bootstrap.cc:492] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d: No bootstrap required, opened a new log
I20260923 17:26:30.723840 14701 ts_tablet_manager.cc:1397] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260923 17:26:30.724327 14704 log.cc:824] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:30.725776 14702 tablet_bootstrap.cc:492] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336: No bootstrap required, opened a new log
I20260923 17:26:30.726059 14702 ts_tablet_manager.cc:1397] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 17:26:30.727038 14704 tablet_bootstrap.cc:492] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667: No bootstrap required, opened a new log
I20260923 17:26:30.727321 14704 ts_tablet_manager.cc:1397] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260923 17:26:30.734973 14701 raft_consensus.cc:348] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe936d76aed4bb1b82aee1f91c48336" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44325 } } peers { permanent_uuid: "680b92d061514308991709809730056d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38227 } } peers { permanent_uuid: "5b060851eac84964bf69242d77574667" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36833 } }
I20260923 17:26:30.735563 14701 raft_consensus.cc:374] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:30.735723 14701 raft_consensus.cc:729] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 680b92d061514308991709809730056d, State: Initialized, Role: FOLLOWER
I20260923 17:26:30.736446 14701 consensus_queue.cc:260] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe936d76aed4bb1b82aee1f91c48336" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44325 } } peers { permanent_uuid: "680b92d061514308991709809730056d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38227 } } peers { permanent_uuid: "5b060851eac84964bf69242d77574667" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36833 } }
I20260923 17:26:30.736430 14704 raft_consensus.cc:348] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe936d76aed4bb1b82aee1f91c48336" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44325 } } peers { permanent_uuid: "680b92d061514308991709809730056d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38227 } } peers { permanent_uuid: "5b060851eac84964bf69242d77574667" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36833 } }
I20260923 17:26:30.736790 14704 raft_consensus.cc:374] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:30.736932 14704 raft_consensus.cc:729] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b060851eac84964bf69242d77574667, State: Initialized, Role: FOLLOWER
I20260923 17:26:30.737200 14702 raft_consensus.cc:348] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe936d76aed4bb1b82aee1f91c48336" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44325 } } peers { permanent_uuid: "680b92d061514308991709809730056d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38227 } } peers { permanent_uuid: "5b060851eac84964bf69242d77574667" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36833 } }
I20260923 17:26:30.737709 14702 raft_consensus.cc:374] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:30.737846 14702 raft_consensus.cc:729] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe936d76aed4bb1b82aee1f91c48336, State: Initialized, Role: FOLLOWER
I20260923 17:26:30.737735 14704 consensus_queue.cc:260] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe936d76aed4bb1b82aee1f91c48336" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44325 } } peers { permanent_uuid: "680b92d061514308991709809730056d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38227 } } peers { permanent_uuid: "5b060851eac84964bf69242d77574667" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36833 } }
I20260923 17:26:30.738456 14701 ts_tablet_manager.cc:1428] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260923 17:26:30.738608 14702 consensus_queue.cc:260] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe936d76aed4bb1b82aee1f91c48336" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44325 } } peers { permanent_uuid: "680b92d061514308991709809730056d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38227 } } peers { permanent_uuid: "5b060851eac84964bf69242d77574667" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36833 } }
I20260923 17:26:30.739526 14683 heartbeater.cc:503] Master 127.28.72.254:42277 was elected leader, sending a full tablet report...
I20260923 17:26:30.740523 14702 ts_tablet_manager.cc:1428] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260923 17:26:30.740900 14553 heartbeater.cc:503] Master 127.28.72.254:42277 was elected leader, sending a full tablet report...
I20260923 17:26:30.743096 14708 raft_consensus.cc:482] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:26:30.743427 14704 ts_tablet_manager.cc:1428] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20260923 17:26:30.743461 14708 raft_consensus.cc:504] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe936d76aed4bb1b82aee1f91c48336" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44325 } } peers { permanent_uuid: "680b92d061514308991709809730056d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38227 } } peers { permanent_uuid: "5b060851eac84964bf69242d77574667" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36833 } }
I20260923 17:26:30.747392 14708 leader_election.cc:288] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fe936d76aed4bb1b82aee1f91c48336 (127.28.72.194:44325), 680b92d061514308991709809730056d (127.28.72.193:38227)
I20260923 17:26:30.756693 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.746959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:49347 (local address 127.28.72.193:38227)
0923 17:26:30.747235 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:30.747244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:30.747270 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:26:30.747380 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:30.747389 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:30.747513 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:30.747617 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:30.748491 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.749961 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.753642 (+ 3681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.755537 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.755745 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.755844 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:30.756325 (+ 481us) server_negotiation.cc:299] Negotiation successful
0923 17:26:30.756481 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:26:30.756801 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.746115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38227 (local address 127.28.72.195:49347)
0923 17:26:30.746477 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:30.746498 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:30.746626 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:30.747881 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:30.747889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:30.747930 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:30.748221 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:30.748236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.750156 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.750166 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:30.753326 (+ 3160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:30.753379 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.755953 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.755961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:30.756149 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.756176 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:26:30.756260 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:26:30.756432 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260923 17:26:30.758337 14377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf8c3d34f5a462196e045732b804693" candidate_uuid: "5b060851eac84964bf69242d77574667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "680b92d061514308991709809730056d" is_pre_election: true
I20260923 17:26:30.758924 14377 raft_consensus.cc:2454] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b060851eac84964bf69242d77574667 in term 0.
I20260923 17:26:30.759824 14572 leader_election.cc:302] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b060851eac84964bf69242d77574667, 680b92d061514308991709809730056d; no voters:
I20260923 17:26:30.760340 14710 raft_consensus.cc:2781] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:26:30.760509 14710 raft_consensus.cc:482] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:26:30.760663 14710 raft_consensus.cc:3037] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:30.762202 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.747015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:33897 (local address 127.28.72.194:44325)
0923 17:26:30.747290 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:30.747296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:30.755929 (+ 8633us) server_negotiation.cc:407] Connection header received
0923 17:26:30.756145 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:30.756151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:30.756269 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:30.756366 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:30.757115 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.758108 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.760248 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.761271 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.761406 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.761458 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:30.761895 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 17:26:30.762003 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260923 17:26:30.762316 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.746339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:44325 (local address 127.28.72.195:33897)
0923 17:26:30.755798 (+ 9459us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:30.755819 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:30.755995 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:30.756416 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:30.756421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:30.756450 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:30.756839 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:30.756863 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.758349 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.758354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:30.760031 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:30.760050 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.761467 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.761473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:30.761657 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.761692 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:26:30.761808 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 17:26:30.761971 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9216,"thread_start_us":8952,"threads_started":1}
I20260923 17:26:30.763557 14508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf8c3d34f5a462196e045732b804693" candidate_uuid: "5b060851eac84964bf69242d77574667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe936d76aed4bb1b82aee1f91c48336" is_pre_election: true
I20260923 17:26:30.763968 14710 raft_consensus.cc:504] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe936d76aed4bb1b82aee1f91c48336" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44325 } } peers { permanent_uuid: "680b92d061514308991709809730056d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38227 } } peers { permanent_uuid: "5b060851eac84964bf69242d77574667" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36833 } }
I20260923 17:26:30.764089 14508 raft_consensus.cc:2454] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b060851eac84964bf69242d77574667 in term 0.
I20260923 17:26:30.764734 14710 leader_election.cc:288] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [CANDIDATE]: Term 1 election: Requested vote from peers 7fe936d76aed4bb1b82aee1f91c48336 (127.28.72.194:44325), 680b92d061514308991709809730056d (127.28.72.193:38227)
I20260923 17:26:30.765290 14508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf8c3d34f5a462196e045732b804693" candidate_uuid: "5b060851eac84964bf69242d77574667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe936d76aed4bb1b82aee1f91c48336"
I20260923 17:26:30.765338 14377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf8c3d34f5a462196e045732b804693" candidate_uuid: "5b060851eac84964bf69242d77574667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "680b92d061514308991709809730056d"
I20260923 17:26:30.765539 14508 raft_consensus.cc:3037] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:30.765551 14377 raft_consensus.cc:3037] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:30.769029 14508 raft_consensus.cc:2454] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b060851eac84964bf69242d77574667 in term 1.
I20260923 17:26:30.769032 14377 raft_consensus.cc:2454] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b060851eac84964bf69242d77574667 in term 1.
I20260923 17:26:30.769606 14572 leader_election.cc:302] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b060851eac84964bf69242d77574667, 680b92d061514308991709809730056d; no voters:
I20260923 17:26:30.769927 14710 raft_consensus.cc:2781] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:30.770169 14710 raft_consensus.cc:686] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [term 1 LEADER]: Becoming Leader. State: Replica: 5b060851eac84964bf69242d77574667, State: Running, Role: LEADER
I20260923 17:26:30.770727 14710 consensus_queue.cc:237] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe936d76aed4bb1b82aee1f91c48336" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44325 } } peers { permanent_uuid: "680b92d061514308991709809730056d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38227 } } peers { permanent_uuid: "5b060851eac84964bf69242d77574667" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36833 } }
I20260923 17:26:30.776197 14235 catalog_manager.cc:5495] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b060851eac84964bf69242d77574667 (127.28.72.195). New cstate: current_term: 1 leader_uuid: "5b060851eac84964bf69242d77574667" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe936d76aed4bb1b82aee1f91c48336" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "680b92d061514308991709809730056d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b060851eac84964bf69242d77574667" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36833 } health_report { overall_health: HEALTHY } } }
I20260923 17:26:30.816591 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.810604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.28.72.195:36833)
0923 17:26:30.810849 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:30.810854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:30.811045 (+ 191us) server_negotiation.cc:407] Connection header received
0923 17:26:30.811399 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:30.811405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:30.811501 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:30.811651 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:26:30.812055 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.812792 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.814266 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.814713 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.814818 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.815466 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:26:30.816037 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:26:30.816168 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:30.816308 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 17:26:30.816381 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:26:30.826305 14377 raft_consensus.cc:1261] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d [term 1 FOLLOWER]: Refusing update from remote peer 5b060851eac84964bf69242d77574667: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:26:30.826305 14508 raft_consensus.cc:1261] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336 [term 1 FOLLOWER]: Refusing update from remote peer 5b060851eac84964bf69242d77574667: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:26:30.827708 14708 consensus_queue.cc:1035] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe936d76aed4bb1b82aee1f91c48336" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:26:30.828114 14710 consensus_queue.cc:1035] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [LEADER]: Connected to new peer: Peer: permanent_uuid: "680b92d061514308991709809730056d" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 38227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:26:30.847533 14715 mvcc.cc:205] Tried to move back new op lower bound from 7332595264812724224 to 7332595264601583616. Current Snapshot: MvccSnapshot[applied={T|T < 7332595264812724224 or (T in {7332595264812724224})}]
I20260923 17:26:30.855315 14235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:26:30.855684 14235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:26:30.859665 14235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a936bc3c5f548a3a105bf45ae77e60b: Sending DeleteTablet for 3 replicas of tablet acf8c3d34f5a462196e045732b804693
I20260923 17:26:30.861366 14618 tablet_service.cc:1514] Processing DeleteTablet for tablet acf8c3d34f5a462196e045732b804693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:30 UTC) from {username='slave'} at 127.0.0.1:48814
I20260923 17:26:30.862259 14488 tablet_service.cc:1514] Processing DeleteTablet for tablet acf8c3d34f5a462196e045732b804693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:30 UTC) from {username='slave'} at 127.0.0.1:46224
I20260923 17:26:30.862468 14723 tablet_replica.cc:330] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667: stopping tablet replica
I20260923 17:26:30.862934 14723 raft_consensus.cc:2229] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:26:30.863070 14357 tablet_service.cc:1514] Processing DeleteTablet for tablet acf8c3d34f5a462196e045732b804693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:30 UTC) from {username='slave'} at 127.0.0.1:39164
I20260923 17:26:30.863291 14724 tablet_replica.cc:330] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336: stopping tablet replica
I20260923 17:26:30.863709 14724 raft_consensus.cc:2229] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:26:30.863768 14723 raft_consensus.cc:2258] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:30.864583 14724 raft_consensus.cc:2258] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:30.865938 14723 ts_tablet_manager.cc:1905] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:30.867281 14724 ts_tablet_manager.cc:1905] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:30.867997 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 14295
I20260923 17:26:30.874990 14723 ts_tablet_manager.cc:1918] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:30.875382 14723 log.cc:1196] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/acf8c3d34f5a462196e045732b804693
I20260923 17:26:30.875998 14724 ts_tablet_manager.cc:1918] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:30.876120 14723 ts_tablet_manager.cc:1939] T acf8c3d34f5a462196e045732b804693 P 5b060851eac84964bf69242d77574667: Deleting consensus metadata
I20260923 17:26:30.876379 14724 log.cc:1196] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/acf8c3d34f5a462196e045732b804693
I20260923 17:26:30.877094 14724 ts_tablet_manager.cc:1939] T acf8c3d34f5a462196e045732b804693 P 7fe936d76aed4bb1b82aee1f91c48336: Deleting consensus metadata
I20260923 17:26:30.878178 14220 catalog_manager.cc:4841] TS 5b060851eac84964bf69242d77574667 (127.28.72.195:36833): tablet acf8c3d34f5a462196e045732b804693 (table test-table [id=f7c72fa250944d94a50dd5d3069b927a]) successfully deleted
I20260923 17:26:30.879344 14222 catalog_manager.cc:4841] TS 7fe936d76aed4bb1b82aee1f91c48336 (127.28.72.194:44325): tablet acf8c3d34f5a462196e045732b804693 (table test-table [id=f7c72fa250944d94a50dd5d3069b927a]) successfully deleted
I20260923 17:26:30.880393 14725 tablet_replica.cc:330] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d: stopping tablet replica
I20260923 17:26:30.880959 14725 raft_consensus.cc:2229] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:26:30.881592 14725 raft_consensus.cc:2258] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:30.883859 14725 ts_tablet_manager.cc:1905] T acf8c3d34f5a462196e045732b804693 P 680b92d061514308991709809730056d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:30.888053 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:30.869435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.28.72.193:38227)
0923 17:26:30.869628 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:30.869634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:30.879683 (+ 10049us) server_negotiation.cc:407] Connection header received
0923 17:26:30.879904 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:30.879914 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:30.880117 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:30.880280 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:30.880793 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.881745 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.883830 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:30.884370 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:30.884454 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:30.886052 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:30.886062 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:30.886076 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:30.887098 (+ 1022us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:30.887212 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:30.887245 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:30.887375 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:30.887504 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 17:26:30.887623 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:26:30.888795 14417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:30.981592 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 14295
W20260923 17:26:31.006249 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.72.193:38227 (error 108)
I20260923 17:26:31.006469 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 14426
I20260923 17:26:31.008652 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:31.008055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38227 (local address 127.0.0.1:39182)
0923 17:26:31.008337 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:31.008465 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260923 17:26:31.008924 14221 catalog_manager.cc:4584] TS 680b92d061514308991709809730056d (127.28.72.193:38227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet acf8c3d34f5a462196e045732b804693: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38227: connect: Connection refused (error 111)
I20260923 17:26:31.014771 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:31.007536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.28.72.194:44325)
0923 17:26:31.007774 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:31.007781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:31.008141 (+ 360us) server_negotiation.cc:407] Connection header received
0923 17:26:31.008339 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:31.008346 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:31.008492 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:31.008603 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:31.009158 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:31.010048 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:31.011526 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:31.011953 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:31.012095 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:31.013296 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:31.013306 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:31.013321 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:31.013933 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:31.013990 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:31.014012 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:31.014144 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:31.014309 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 17:26:31.014429 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:26:31.015570 14548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:31.062698 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:31.062113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38227 (local address 127.0.0.1:39190)
0923 17:26:31.062401 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:31.062502 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260923 17:26:31.064072 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:31.063622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38227 (local address 127.0.0.1:39200)
0923 17:26:31.063821 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:31.063901 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260923 17:26:31.110975 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 14426
I20260923 17:26:31.125135 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:31.124471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38227 (local address 127.0.0.1:39214)
0923 17:26:31.124756 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:31.124897 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 17:26:31.126457 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:31.126001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38227 (local address 127.0.0.1:39226)
0923 17:26:31.126192 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:31.126287 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 17:26:31.129297 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 14556
I20260923 17:26:31.137346 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:31.130291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48824 (local address 127.28.72.195:36833)
0923 17:26:31.130493 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:31.130499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:31.130531 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:26:31.130673 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:31.130680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:31.130760 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:31.130898 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:31.131406 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:31.132255 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:31.133646 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:31.134323 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:31.134481 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:31.135763 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:31.135773 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:31.135787 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:31.136451 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:31.136506 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:31.136536 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:31.136628 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:31.136826 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 17:26:31.136914 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:26:31.138191 14678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:31.207424 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:31.206853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38227 (local address 127.0.0.1:39238)
0923 17:26:31.207099 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:31.207261 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260923 17:26:31.208551 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:31.208144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:38227 (local address 127.0.0.1:39250)
0923 17:26:31.208360 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:31.208435 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.193:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260923 17:26:31.262241 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 14556
I20260923 17:26:31.282235 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 14205
I20260923 17:26:31.283609 14266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:31.397781 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 14205
2026-09-23T17:26:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2888 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:26:31 dist-test-slave-j0m4 krb5kdc[14734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:26:31 dist-test-slave-j0m4 krb5kdc[14734](info): set up 2 sockets
Sep 23 17:26:31 dist-test-slave-j0m4 krb5kdc[14734](info): commencing operation
krb5kdc: starting...
W20260923 17:26:32.539538 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.003s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:26:32 dist-test-slave-j0m4 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:26:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:26:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:26:32.810806 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:37423
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:37423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:33.159822 14747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:33.160209 14747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:33.160387 14747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:33.168099 14747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:26:33.168263 14747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:33.168385 14747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:33.168488 14747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:26:33.168577 14747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:26:33.179020 14747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:37423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:37423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:33.180506 14747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:33.181874 14747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:33.194659 14752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:33.194919 14753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:33.195299 14755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:33.196084 14747 server_base.cc:1034] running on GCE node
I20260923 17:26:33.197000 14747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:33.198614 14747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:33.199839 14747 hybrid_clock.cc:648] HybridClock initialized: now 1790184393199781 us; error 52 us; skew 500 ppm
I20260923 17:26:33.200229 14747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:33 dist-test-slave-j0m4 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184393, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:33.203716 14747 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:26:33.205574 14747 webserver.cc:460] Webserver started at http://127.28.72.254:33515/ using document root <none> and password file <none>
I20260923 17:26:33.206317 14747 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:33.206462 14747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:33.206807 14747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:33.210053 14747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "dc8a5ce4c083460682b8bceb7e0b7a59"
format_stamp: "Formatted at 2026-09-23 17:26:33 on dist-test-slave-j0m4"
server_key: "af8b3aa2b4cb7465869ff578d8834c74"
server_key_iv: "a7d0ca4cb71bb0d39bf15811b6e15b30"
server_key_version: "encryptionkey@0"
I20260923 17:26:33.210856 14747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "dc8a5ce4c083460682b8bceb7e0b7a59"
format_stamp: "Formatted at 2026-09-23 17:26:33 on dist-test-slave-j0m4"
server_key: "af8b3aa2b4cb7465869ff578d8834c74"
server_key_iv: "a7d0ca4cb71bb0d39bf15811b6e15b30"
server_key_version: "encryptionkey@0"
I20260923 17:26:33.216557 14747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:26:33.220609 14762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:33.222167 14747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:26:33.222352 14747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "dc8a5ce4c083460682b8bceb7e0b7a59"
format_stamp: "Formatted at 2026-09-23 17:26:33 on dist-test-slave-j0m4"
server_key: "af8b3aa2b4cb7465869ff578d8834c74"
server_key_iv: "a7d0ca4cb71bb0d39bf15811b6e15b30"
server_key_version: "encryptionkey@0"
I20260923 17:26:33.222620 14747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:33.244088 14747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:33.248971 14747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:33.249325 14747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:33.270032 14747 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:37423
I20260923 17:26:33.270087 14813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:37423 every 8 connection(s)
I20260923 17:26:33.271605 14747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:26:33.276072 14814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:33.277283 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 14747
I20260923 17:26:33.277534 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:26:33.278198 28963 external_mini_cluster.cc:1428] Setting key 85a110889ee15e4facb5df52f2a9665e
I20260923 17:26:33.292968 14814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59: Bootstrap starting.
Sep 23 17:26:33 dist-test-slave-j0m4 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:26:33.297317 14814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:33.298878 14814 log.cc:824] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:33.302771 14814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59: No bootstrap required, opened a new log
I20260923 17:26:33.306810 14817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:33.280539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38114 (local address 127.28.72.254:37423)
0923 17:26:33.281550 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:33.281565 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:33.281617 (+ 52us) server_negotiation.cc:407] Connection header received
0923 17:26:33.283755 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:33.283789 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:33.284350 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:33.284834 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:33.287037 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:33.288729 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:33.290849 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:33.291767 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:33.291951 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:33.297254 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:33.297268 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:33.297299 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:33.297360 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:33.301533 (+ 4173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:33.302464 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:33.302473 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:33.302481 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:33.302702 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:33.303196 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:33.303201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:33.303204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:33.303997 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:33.304309 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:33.304767 (+ 458us) server_negotiation.cc:299] Negotiation successful
0923 17:26:33.305294 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":678,"thread_start_us":436,"threads_started":1}
I20260923 17:26:33.312651 14814 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc8a5ce4c083460682b8bceb7e0b7a59" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37423 } }
I20260923 17:26:33.313036 14814 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:33.313117 14814 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc8a5ce4c083460682b8bceb7e0b7a59, State: Initialized, Role: FOLLOWER
I20260923 17:26:33.313741 14814 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc8a5ce4c083460682b8bceb7e0b7a59" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37423 } }
I20260923 17:26:33.313967 14814 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:26:33.314069 14814 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:26:33.314198 14814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:33.317021 14814 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc8a5ce4c083460682b8bceb7e0b7a59" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37423 } }
I20260923 17:26:33.317556 14814 leader_election.cc:302] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc8a5ce4c083460682b8bceb7e0b7a59; no voters:
I20260923 17:26:33.318210 14814 leader_election.cc:288] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:26:33.318430 14819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:33.319276 14819 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [term 1 LEADER]: Becoming Leader. State: Replica: dc8a5ce4c083460682b8bceb7e0b7a59, State: Running, Role: LEADER
I20260923 17:26:33.319931 14819 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc8a5ce4c083460682b8bceb7e0b7a59" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37423 } }
I20260923 17:26:33.320633 14814 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:26:33.322926 14821 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc8a5ce4c083460682b8bceb7e0b7a59. Latest consensus state: current_term: 1 leader_uuid: "dc8a5ce4c083460682b8bceb7e0b7a59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc8a5ce4c083460682b8bceb7e0b7a59" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37423 } } }
I20260923 17:26:33.322988 14820 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc8a5ce4c083460682b8bceb7e0b7a59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc8a5ce4c083460682b8bceb7e0b7a59" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37423 } } }
I20260923 17:26:33.323433 14821 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:33.323472 14820 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:33.325829 14824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:26:33.331686 14824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:26:33.344256 14824 catalog_manager.cc:1347] Generated new cluster ID: 513554edbfdb4af6b5c53ffc67268e76
I20260923 17:26:33.344420 14824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:26:33.370728 14824 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:26:33.372313 14824 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:26:33.384608 14824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59: Generated new TSK 0
I20260923 17:26:33.385563 14824 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:26:33.455439 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37423
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:33.804939 14840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:33.805399 14840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:33.805541 14840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:33.814005 14840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:33.814132 14840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:33.814263 14840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:26:33.824915 14840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:33.826272 14840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:33.827816 14840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:33.842060 14848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:33.842114 14845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:33.842113 14846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:33.842584 14840 server_base.cc:1034] running on GCE node
I20260923 17:26:33.843477 14840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:33.844524 14840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:33.845784 14840 hybrid_clock.cc:648] HybridClock initialized: now 1790184393845727 us; error 48 us; skew 500 ppm
I20260923 17:26:33.846146 14840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:33 dist-test-slave-j0m4 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184393, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:33.849656 14840 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:26:33.851837 14840 webserver.cc:460] Webserver started at http://127.28.72.193:35319/ using document root <none> and password file <none>
I20260923 17:26:33.852573 14840 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:33.852705 14840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:33.853041 14840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:33.856144 14840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "54d96bf95c2245f6add7865c4f348e6a"
format_stamp: "Formatted at 2026-09-23 17:26:33 on dist-test-slave-j0m4"
server_key: "4f5a11e6e1fbee459268e68eafd9ed47"
server_key_iv: "34436b338f73fbf1d2a6bbcd799d7fed"
server_key_version: "encryptionkey@0"
I20260923 17:26:33.856942 14840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "54d96bf95c2245f6add7865c4f348e6a"
format_stamp: "Formatted at 2026-09-23 17:26:33 on dist-test-slave-j0m4"
server_key: "4f5a11e6e1fbee459268e68eafd9ed47"
server_key_iv: "34436b338f73fbf1d2a6bbcd799d7fed"
server_key_version: "encryptionkey@0"
I20260923 17:26:33.862530 14840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 17:26:33.866453 14855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:33.868168 14840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:26:33.868322 14840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "54d96bf95c2245f6add7865c4f348e6a"
format_stamp: "Formatted at 2026-09-23 17:26:33 on dist-test-slave-j0m4"
server_key: "4f5a11e6e1fbee459268e68eafd9ed47"
server_key_iv: "34436b338f73fbf1d2a6bbcd799d7fed"
server_key_version: "encryptionkey@0"
I20260923 17:26:33.868569 14840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:33.881621 14840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:33.886984 14840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:33.887380 14840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:33.888835 14840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:33.891018 14840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:33.891196 14840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260923 17:26:33.891355 14840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:33.891417 14840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:33.925833 14840 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:46879
I20260923 17:26:33.925882 14967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:46879 every 8 connection(s)
I20260923 17:26:33.927274 14840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:26:33.934954 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 14840
I20260923 17:26:33.935206 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:26:33.935771 28963 external_mini_cluster.cc:1428] Setting key 65703bcccbd1c46fb842cca485f3c76d
Sep 23 17:26:33 dist-test-slave-j0m4 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184393, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:26:33.948419 14970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:33.929942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:43973 (local address 127.28.72.254:37423)
0923 17:26:33.930396 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:33.930401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:33.931439 (+ 1038us) server_negotiation.cc:407] Connection header received
0923 17:26:33.933371 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:33.933377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:33.933486 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:33.933603 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:33.935268 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:33.936348 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:33.938049 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:33.938497 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:33.938644 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:33.942383 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:33.942388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:33.942391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:33.942427 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:33.944992 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:33.945795 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:33.945800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:33.945802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:33.945935 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:33.946493 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:33.946497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:33.946499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:33.946789 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:33.946916 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:33.947963 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0923 17:26:33.948154 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":222,"threads_started":1}
I20260923 17:26:33.950030 14971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:33.930533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37423 (local address 127.28.72.193:43973)
0923 17:26:33.931282 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:33.931328 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:33.933104 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:33.933846 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:33.933855 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:33.934458 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:33.935037 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:33.935056 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:33.936537 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:33.936543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:33.937882 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:33.937895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:33.938758 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:33.938767 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:33.939037 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:33.939845 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:26:33.939868 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:33.942139 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:33.945155 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:33.945165 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:33.945187 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:33.945654 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:33.946045 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:33.946050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:33.946054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:33.946342 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:33.946930 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:33.946942 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:33.947513 (+ 571us) client_negotiation.cc:769] Sending connection context
0923 17:26:33.947835 (+ 322us) client_negotiation.cc:240] Negotiation successful
0923 17:26:33.948482 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":261,"threads_started":1}
I20260923 17:26:33.951849 14968 heartbeater.cc:347] Connected to a master server at 127.28.72.254:37423
I20260923 17:26:33.952204 14968 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:33.953114 14968 heartbeater.cc:511] Master 127.28.72.254:37423 requested a full tablet report, sending...
I20260923 17:26:33.955699 14779 ts_manager.cc:194] Registered new tserver with Master: 54d96bf95c2245f6add7865c4f348e6a (127.28.72.193:46879)
I20260923 17:26:33.957285 14779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:43973
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:26:33.999205 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37423
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:34.352828 14974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:34.353133 14974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:34.353288 14974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:34.360818 14974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:34.360949 14974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:34.361125 14974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:26:34.371622 14974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:34.373031 14974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:34.374378 14974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:34.388011 14979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:34.388070 14982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:34.388046 14980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:34.388957 14974 server_base.cc:1034] running on GCE node
I20260923 17:26:34.389719 14974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:34.390869 14974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:34.392161 14974 hybrid_clock.cc:648] HybridClock initialized: now 1790184394392083 us; error 66 us; skew 500 ppm
I20260923 17:26:34.392541 14974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:34 dist-test-slave-j0m4 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184394, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:34.395889 14974 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:26:34.397437 14974 webserver.cc:460] Webserver started at http://127.28.72.194:36599/ using document root <none> and password file <none>
I20260923 17:26:34.398141 14974 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:34.398285 14974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:34.398607 14974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:34.401712 14974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "d6235b352c0940a0bb459768fafb2e81"
format_stamp: "Formatted at 2026-09-23 17:26:34 on dist-test-slave-j0m4"
server_key: "22a2a0865335000b9d9b8289d852ec4e"
server_key_iv: "29dc767a5d0750a21d0b5f7cc7e64bbc"
server_key_version: "encryptionkey@0"
I20260923 17:26:34.402522 14974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "d6235b352c0940a0bb459768fafb2e81"
format_stamp: "Formatted at 2026-09-23 17:26:34 on dist-test-slave-j0m4"
server_key: "22a2a0865335000b9d9b8289d852ec4e"
server_key_iv: "29dc767a5d0750a21d0b5f7cc7e64bbc"
server_key_version: "encryptionkey@0"
I20260923 17:26:34.408064 14974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 17:26:34.411752 14989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:34.413189 14974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:26:34.413388 14974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "d6235b352c0940a0bb459768fafb2e81"
format_stamp: "Formatted at 2026-09-23 17:26:34 on dist-test-slave-j0m4"
server_key: "22a2a0865335000b9d9b8289d852ec4e"
server_key_iv: "29dc767a5d0750a21d0b5f7cc7e64bbc"
server_key_version: "encryptionkey@0"
I20260923 17:26:34.413609 14974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:34.436069 14974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:34.440901 14974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:34.441169 14974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:34.442240 14974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:34.443694 14974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:34.443779 14974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:34.443866 14974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:34.443910 14974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:34.478169 14974 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:43071
I20260923 17:26:34.478281 15101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:43071 every 8 connection(s)
I20260923 17:26:34.479652 14974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:26:34.485415 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 14974
I20260923 17:26:34.485598 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:26:34.486047 28963 external_mini_cluster.cc:1428] Setting key 08888aac791f2a21b7b1a8a3f278c664
Sep 23 17:26:34 dist-test-slave-j0m4 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184394, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:26:34.501284 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:34.482586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:42253 (local address 127.28.72.254:37423)
0923 17:26:34.483035 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:34.483040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:34.484122 (+ 1082us) server_negotiation.cc:407] Connection header received
0923 17:26:34.486060 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:34.486066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:34.486188 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:34.486316 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:34.488171 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:34.488885 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:34.490461 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:34.490895 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:34.491025 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:34.494530 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:34.494535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:34.494538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:34.494571 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:34.497410 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:34.498520 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:34.498524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:34.498527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:34.498673 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:34.499226 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:34.499230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:34.499232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:34.499558 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:34.499684 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:34.500782 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0923 17:26:34.500995 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":224,"threads_started":1}
I20260923 17:26:34.502725 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:34.483232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37423 (local address 127.28.72.194:42253)
0923 17:26:34.483935 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:34.483973 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:34.485785 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:34.486606 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:34.486621 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:34.487292 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:34.487981 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:34.488006 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:34.489047 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:34.489052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:34.490300 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:34.490312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:34.491158 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:34.491168 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:34.491406 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:34.492114 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:26:34.492141 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:34.494289 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:34.497583 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:34.497597 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:34.497620 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:34.498351 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:34.498792 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:34.498800 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:34.498807 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:34.499077 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:34.499704 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:34.499714 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:34.500213 (+ 499us) client_negotiation.cc:769] Sending connection context
0923 17:26:34.500536 (+ 323us) client_negotiation.cc:240] Negotiation successful
0923 17:26:34.501172 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":260,"threads_started":1}
I20260923 17:26:34.504401 15102 heartbeater.cc:347] Connected to a master server at 127.28.72.254:37423
I20260923 17:26:34.504747 15102 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:34.505539 15102 heartbeater.cc:511] Master 127.28.72.254:37423 requested a full tablet report, sending...
I20260923 17:26:34.507216 14779 ts_manager.cc:194] Registered new tserver with Master: d6235b352c0940a0bb459768fafb2e81 (127.28.72.194:43071)
I20260923 17:26:34.508170 14779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:42253
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:26:34.547549 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37423
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:34.894141 15108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:34.894647 15108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:34.894814 15108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:34.902628 15108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:34.902812 15108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:34.902992 15108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:26:34.913348 15108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:34.914749 15108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:34.916150 15108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:34.930500 15114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:34.930537 15113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:34.931483 15116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:34.934019 15108 server_base.cc:1034] running on GCE node
I20260923 17:26:34.934762 15108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:34.935945 15108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:34.937192 15108 hybrid_clock.cc:648] HybridClock initialized: now 1790184394937125 us; error 55 us; skew 500 ppm
I20260923 17:26:34.937577 15108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:34 dist-test-slave-j0m4 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184394, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:34.941344 15108 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:26:34.943470 15108 webserver.cc:460] Webserver started at http://127.28.72.195:36165/ using document root <none> and password file <none>
I20260923 17:26:34.944270 15108 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:34.944422 15108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:34.944761 15108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:34.947986 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "bd8bfee4d6aa442993aaef2327c34432"
format_stamp: "Formatted at 2026-09-23 17:26:34 on dist-test-slave-j0m4"
server_key: "0ff5ade494f1cde8c8076b2729111430"
server_key_iv: "41ec0fb3cca0bb88af08a6a47d6588c1"
server_key_version: "encryptionkey@0"
I20260923 17:26:34.948838 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "bd8bfee4d6aa442993aaef2327c34432"
format_stamp: "Formatted at 2026-09-23 17:26:34 on dist-test-slave-j0m4"
server_key: "0ff5ade494f1cde8c8076b2729111430"
server_key_iv: "41ec0fb3cca0bb88af08a6a47d6588c1"
server_key_version: "encryptionkey@0"
I20260923 17:26:34.954715 15108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:26:34.958920 15123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:34.960588 15108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 17:26:34.960795 15108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "bd8bfee4d6aa442993aaef2327c34432"
format_stamp: "Formatted at 2026-09-23 17:26:34 on dist-test-slave-j0m4"
server_key: "0ff5ade494f1cde8c8076b2729111430"
server_key_iv: "41ec0fb3cca0bb88af08a6a47d6588c1"
server_key_version: "encryptionkey@0"
I20260923 17:26:34.961056 15108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:34.961548 14968 heartbeater.cc:503] Master 127.28.72.254:37423 was elected leader, sending a full tablet report...
I20260923 17:26:34.989566 15108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:34.994491 15108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:34.994761 15108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:34.995826 15108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:34.997335 15108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:34.997416 15108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:34.997509 15108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:34.997555 15108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:35.033253 15108 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:35177
I20260923 17:26:35.033330 15235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:35177 every 8 connection(s)
I20260923 17:26:35.034680 15108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:26:35.044438 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 15108
I20260923 17:26:35.044618 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:26:35.045063 28963 external_mini_cluster.cc:1428] Setting key 25df87cebedbe7c2e22d410d033b3e1a
Sep 23 17:26:35 dist-test-slave-j0m4 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184394, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:26:35.057572 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.037636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:53059 (local address 127.28.72.254:37423)
0923 17:26:35.038109 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:35.038114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:35.039146 (+ 1032us) server_negotiation.cc:407] Connection header received
0923 17:26:35.041230 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:35.041236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:35.041352 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:35.041470 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:35.043149 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.043976 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.046179 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.046922 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.047066 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.051381 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:35.051388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:35.051392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:35.051438 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:35.054337 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:35.055246 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:35.055251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:35.055253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:35.055395 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:35.055864 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:35.055868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:35.055870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:35.056157 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:35.056281 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:35.057075 (+ 794us) server_negotiation.cc:299] Negotiation successful
0923 17:26:35.057265 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":228,"threads_started":1}
I20260923 17:26:35.058624 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.038219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37423 (local address 127.28.72.195:53059)
0923 17:26:35.038949 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:35.038995 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:35.040934 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:35.041716 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:35.041744 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:35.042324 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:35.042935 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:35.042954 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.044299 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.044304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:35.045996 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:35.046016 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.047258 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.047266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:35.047623 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.048477 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:26:35.048513 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:35.051072 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:35.054515 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:35.054524 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:35.054539 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:35.055077 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:35.055510 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:35.055513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:35.055515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:35.055730 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:35.056304 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:35.056310 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:35.056697 (+ 387us) client_negotiation.cc:769] Sending connection context
0923 17:26:35.056971 (+ 274us) client_negotiation.cc:240] Negotiation successful
0923 17:26:35.057406 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":269,"threads_started":1}
I20260923 17:26:35.060297 15236 heartbeater.cc:347] Connected to a master server at 127.28.72.254:37423
I20260923 17:26:35.060671 15236 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:35.061537 15236 heartbeater.cc:511] Master 127.28.72.254:37423 requested a full tablet report, sending...
I20260923 17:26:35.063063 14778 ts_manager.cc:194] Registered new tserver with Master: bd8bfee4d6aa442993aaef2327c34432 (127.28.72.195:35177)
I20260923 17:26:35.064101 14778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:53059
I20260923 17:26:35.074759 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:26:35.094645 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.079457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.28.72.254:37423)
0923 17:26:35.079647 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:35.079652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:35.080315 (+ 663us) server_negotiation.cc:407] Connection header received
0923 17:26:35.080396 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:35.080400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:35.080473 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:35.080574 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:35.082734 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.083826 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.085404 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.085889 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.086006 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.088951 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:35.088958 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:35.088967 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:35.089009 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:35.091910 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:35.092784 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:35.092788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:35.092791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:35.092923 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:35.093335 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:35.093338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:35.093340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:35.093658 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:35.093885 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:35.094201 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 17:26:35.094357 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:26:35.099356 14778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:26:35.107102 14778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:26:35.131763 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.121291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.28.72.194:43071)
0923 17:26:35.122127 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:35.122139 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:35.122692 (+ 553us) server_negotiation.cc:407] Connection header received
0923 17:26:35.123014 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:35.123023 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:35.123300 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:35.123482 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:35.124563 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.126274 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.128813 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.130304 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.130455 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.130675 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:35.131267 (+ 592us) server_negotiation.cc:299] Negotiation successful
0923 17:26:35.131442 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":1579,"threads_started":1}
I20260923 17:26:35.131862 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.120678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:43071 (local address 127.0.0.1:38728)
0923 17:26:35.122583 (+ 1905us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:35.122600 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:35.122795 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:35.123724 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:35.123738 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:35.123782 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:35.124352 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:35.124366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.126443 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.126448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:35.128632 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:35.128647 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.130816 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.130824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:35.130998 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.131034 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:26:35.131211 (+ 177us) client_negotiation.cc:240] Negotiation successful
0923 17:26:35.131436 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1781,"mutex_wait_us":1,"thread_start_us":925,"threads_started":1}
I20260923 17:26:35.132622 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.120270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:46879 (local address 127.0.0.1:57132)
0923 17:26:35.122045 (+ 1775us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:35.122084 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:35.122316 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:35.124926 (+ 2610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:35.124931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:35.124956 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:35.125293 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:35.125305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.127272 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.127278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:35.129402 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:35.129417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.131679 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.131685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:35.131882 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.131914 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:26:35.132014 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 17:26:35.132154 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1523,"mutex_wait_us":24,"thread_start_us":1958,"threads_started":1}
I20260923 17:26:35.132984 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.120615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.28.72.193:46879)
0923 17:26:35.121437 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:35.121449 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:35.122545 (+ 1096us) server_negotiation.cc:407] Connection header received
0923 17:26:35.122665 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:35.122675 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:35.124676 (+ 2001us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:35.124888 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:35.125471 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.127079 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.129581 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.131167 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.131309 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.131533 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:35.132453 (+ 920us) server_negotiation.cc:299] Negotiation successful
0923 17:26:35.132676 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":743,"threads_started":1}
I20260923 17:26:35.134456 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.123158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.28.72.195:35177)
0923 17:26:35.124992 (+ 1834us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:35.125012 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:35.125047 (+ 35us) server_negotiation.cc:407] Connection header received
0923 17:26:35.125140 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:35.125145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:35.125301 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:35.126287 (+ 986us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:35.126608 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.128189 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.130691 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.132844 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.132975 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.133205 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:35.133399 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 17:26:35.133580 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1666,"thread_start_us":1484,"threads_started":1}
I20260923 17:26:35.136370 14903 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc8c28c1aa64ae4857758e1a01be053 (DEFAULT_TABLE table=test-table [id=066d02dd6c4040ae94caf3116b52bf56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:35.136365 15037 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc8c28c1aa64ae4857758e1a01be053 (DEFAULT_TABLE table=test-table [id=066d02dd6c4040ae94caf3116b52bf56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:35.133852 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.122526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:35177 (local address 127.0.0.1:46502)
0923 17:26:35.124234 (+ 1708us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:35.124256 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:35.124370 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:35.125545 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:35.125550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:35.125578 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:35.125933 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:35.125943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.128350 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.128358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:35.130496 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:35.130513 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.133022 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.133028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:35.133202 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.133242 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 17:26:35.133354 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 17:26:35.133483 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1543,"mutex_wait_us":4,"thread_start_us":1339,"threads_started":1}
I20260923 17:26:35.137699 14903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc8c28c1aa64ae4857758e1a01be053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:35.137753 15037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc8c28c1aa64ae4857758e1a01be053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:35.141047 15171 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc8c28c1aa64ae4857758e1a01be053 (DEFAULT_TABLE table=test-table [id=066d02dd6c4040ae94caf3116b52bf56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:35.142750 15171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc8c28c1aa64ae4857758e1a01be053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:35.150453 15255 tablet_bootstrap.cc:492] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81: Bootstrap starting.
I20260923 17:26:35.153602 15255 tablet_bootstrap.cc:654] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:35.155328 15255 log.cc:824] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:35.156147 15256 tablet_bootstrap.cc:492] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a: Bootstrap starting.
I20260923 17:26:35.159328 15258 tablet_bootstrap.cc:492] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432: Bootstrap starting.
I20260923 17:26:35.161032 15256 tablet_bootstrap.cc:654] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:35.162403 15256 log.cc:824] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:35.162586 15258 tablet_bootstrap.cc:654] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:35.163400 15255 tablet_bootstrap.cc:492] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81: No bootstrap required, opened a new log
I20260923 17:26:35.163733 15255 ts_tablet_manager.cc:1397] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20260923 17:26:35.164654 15258 log.cc:824] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:35.166343 15256 tablet_bootstrap.cc:492] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a: No bootstrap required, opened a new log
I20260923 17:26:35.166626 15256 ts_tablet_manager.cc:1397] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 17:26:35.169379 15258 tablet_bootstrap.cc:492] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432: No bootstrap required, opened a new log
I20260923 17:26:35.169667 15258 ts_tablet_manager.cc:1397] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260923 17:26:35.174871 15255 raft_consensus.cc:348] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d96bf95c2245f6add7865c4f348e6a" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46879 } } peers { permanent_uuid: "d6235b352c0940a0bb459768fafb2e81" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 43071 } } peers { permanent_uuid: "bd8bfee4d6aa442993aaef2327c34432" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35177 } }
I20260923 17:26:35.175426 15255 raft_consensus.cc:374] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:35.175527 15255 raft_consensus.cc:729] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6235b352c0940a0bb459768fafb2e81, State: Initialized, Role: FOLLOWER
I20260923 17:26:35.176414 15255 consensus_queue.cc:260] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d96bf95c2245f6add7865c4f348e6a" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46879 } } peers { permanent_uuid: "d6235b352c0940a0bb459768fafb2e81" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 43071 } } peers { permanent_uuid: "bd8bfee4d6aa442993aaef2327c34432" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35177 } }
I20260923 17:26:35.177780 15102 heartbeater.cc:503] Master 127.28.72.254:37423 was elected leader, sending a full tablet report...
I20260923 17:26:35.177703 15256 raft_consensus.cc:348] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d96bf95c2245f6add7865c4f348e6a" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46879 } } peers { permanent_uuid: "d6235b352c0940a0bb459768fafb2e81" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 43071 } } peers { permanent_uuid: "bd8bfee4d6aa442993aaef2327c34432" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35177 } }
I20260923 17:26:35.178192 15256 raft_consensus.cc:374] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:35.178308 15256 raft_consensus.cc:729] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d96bf95c2245f6add7865c4f348e6a, State: Initialized, Role: FOLLOWER
I20260923 17:26:35.178491 15255 ts_tablet_manager.cc:1428] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260923 17:26:35.179015 15256 consensus_queue.cc:260] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d96bf95c2245f6add7865c4f348e6a" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46879 } } peers { permanent_uuid: "d6235b352c0940a0bb459768fafb2e81" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 43071 } } peers { permanent_uuid: "bd8bfee4d6aa442993aaef2327c34432" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35177 } }
I20260923 17:26:35.180994 15256 ts_tablet_manager.cc:1428] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
I20260923 17:26:35.181242 15258 raft_consensus.cc:348] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d96bf95c2245f6add7865c4f348e6a" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46879 } } peers { permanent_uuid: "d6235b352c0940a0bb459768fafb2e81" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 43071 } } peers { permanent_uuid: "bd8bfee4d6aa442993aaef2327c34432" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35177 } }
I20260923 17:26:35.181715 15258 raft_consensus.cc:374] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:35.181813 15258 raft_consensus.cc:729] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8bfee4d6aa442993aaef2327c34432, State: Initialized, Role: FOLLOWER
I20260923 17:26:35.182466 15258 consensus_queue.cc:260] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d96bf95c2245f6add7865c4f348e6a" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46879 } } peers { permanent_uuid: "d6235b352c0940a0bb459768fafb2e81" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 43071 } } peers { permanent_uuid: "bd8bfee4d6aa442993aaef2327c34432" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35177 } }
I20260923 17:26:35.183833 15236 heartbeater.cc:503] Master 127.28.72.254:37423 was elected leader, sending a full tablet report...
I20260923 17:26:35.184743 15258 ts_tablet_manager.cc:1428] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
W20260923 17:26:35.232071 15103 tablet.cc:2369] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:26:35.286464 15237 tablet.cc:2369] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:26:35.321237 15263 raft_consensus.cc:482] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:26:35.321560 15263 raft_consensus.cc:504] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d96bf95c2245f6add7865c4f348e6a" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46879 } } peers { permanent_uuid: "d6235b352c0940a0bb459768fafb2e81" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 43071 } } peers { permanent_uuid: "bd8bfee4d6aa442993aaef2327c34432" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35177 } }
I20260923 17:26:35.323477 15263 leader_election.cc:288] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54d96bf95c2245f6add7865c4f348e6a (127.28.72.193:46879), d6235b352c0940a0bb459768fafb2e81 (127.28.72.194:43071)
I20260923 17:26:35.329798 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.324090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:51765 (local address 127.28.72.194:43071)
0923 17:26:35.324343 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:35.324349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:35.324374 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:26:35.324530 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:35.324538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:35.324634 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:35.324750 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:35.325256 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.326031 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.328007 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.328854 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.329027 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.329108 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:35.329381 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 17:26:35.329513 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260923 17:26:35.329803 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.323816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:43071 (local address 127.28.72.195:51765)
0923 17:26:35.324161 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:35.324185 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:35.324323 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:35.324908 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:35.324913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:35.324941 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:35.325108 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:35.325116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.326162 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.326168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:35.327856 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:35.327869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.329029 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.329035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:35.329219 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.329249 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 17:26:35.329354 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 17:26:35.329505 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260923 17:26:35.330164 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.324082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:41521 (local address 127.28.72.193:46879)
0923 17:26:35.324341 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:35.324348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:35.325023 (+ 675us) server_negotiation.cc:407] Connection header received
0923 17:26:35.325223 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:35.325227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:35.325287 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:35.325374 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:35.325893 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.326671 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.328423 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.329209 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.329410 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.329483 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:35.329754 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 17:26:35.329894 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260923 17:26:35.330417 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.323948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:46879 (local address 127.28.72.195:41521)
0923 17:26:35.324910 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:35.324933 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:35.325078 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:35.325401 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:35.325406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:35.325429 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:35.325749 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:35.325757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.326857 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.326861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:35.328264 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:35.328284 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.329418 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.329424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:35.329588 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.329611 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:26:35.329729 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 17:26:35.330013 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":1,"thread_start_us":579,"threads_started":1}
I20260923 17:26:35.331180 15057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc8c28c1aa64ae4857758e1a01be053" candidate_uuid: "bd8bfee4d6aa442993aaef2327c34432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6235b352c0940a0bb459768fafb2e81" is_pre_election: true
I20260923 17:26:35.331490 14923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc8c28c1aa64ae4857758e1a01be053" candidate_uuid: "bd8bfee4d6aa442993aaef2327c34432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d96bf95c2245f6add7865c4f348e6a" is_pre_election: true
I20260923 17:26:35.331837 15057 raft_consensus.cc:2454] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd8bfee4d6aa442993aaef2327c34432 in term 0.
I20260923 17:26:35.332064 14923 raft_consensus.cc:2454] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd8bfee4d6aa442993aaef2327c34432 in term 0.
I20260923 17:26:35.332615 15126 leader_election.cc:302] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd8bfee4d6aa442993aaef2327c34432, d6235b352c0940a0bb459768fafb2e81; no voters:
I20260923 17:26:35.333004 15263 raft_consensus.cc:2781] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:26:35.333186 15263 raft_consensus.cc:482] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:26:35.333330 15263 raft_consensus.cc:3037] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:35.335768 15263 raft_consensus.cc:504] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d96bf95c2245f6add7865c4f348e6a" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46879 } } peers { permanent_uuid: "d6235b352c0940a0bb459768fafb2e81" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 43071 } } peers { permanent_uuid: "bd8bfee4d6aa442993aaef2327c34432" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35177 } }
I20260923 17:26:35.336457 15263 leader_election.cc:288] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [CANDIDATE]: Term 1 election: Requested vote from peers 54d96bf95c2245f6add7865c4f348e6a (127.28.72.193:46879), d6235b352c0940a0bb459768fafb2e81 (127.28.72.194:43071)
I20260923 17:26:35.336939 14923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc8c28c1aa64ae4857758e1a01be053" candidate_uuid: "bd8bfee4d6aa442993aaef2327c34432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d96bf95c2245f6add7865c4f348e6a"
I20260923 17:26:35.336961 15057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc8c28c1aa64ae4857758e1a01be053" candidate_uuid: "bd8bfee4d6aa442993aaef2327c34432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6235b352c0940a0bb459768fafb2e81"
I20260923 17:26:35.337160 15057 raft_consensus.cc:3037] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:35.337189 14923 raft_consensus.cc:3037] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:35.339527 15057 raft_consensus.cc:2454] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd8bfee4d6aa442993aaef2327c34432 in term 1.
I20260923 17:26:35.339560 14923 raft_consensus.cc:2454] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd8bfee4d6aa442993aaef2327c34432 in term 1.
I20260923 17:26:35.340143 15126 leader_election.cc:302] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 54d96bf95c2245f6add7865c4f348e6a, bd8bfee4d6aa442993aaef2327c34432, d6235b352c0940a0bb459768fafb2e81; no voters:
I20260923 17:26:35.340483 15263 raft_consensus.cc:2781] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:35.340966 15263 raft_consensus.cc:686] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [term 1 LEADER]: Becoming Leader. State: Replica: bd8bfee4d6aa442993aaef2327c34432, State: Running, Role: LEADER
I20260923 17:26:35.341518 15263 consensus_queue.cc:237] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d96bf95c2245f6add7865c4f348e6a" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46879 } } peers { permanent_uuid: "d6235b352c0940a0bb459768fafb2e81" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 43071 } } peers { permanent_uuid: "bd8bfee4d6aa442993aaef2327c34432" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35177 } }
I20260923 17:26:35.347090 14779 catalog_manager.cc:5495] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd8bfee4d6aa442993aaef2327c34432 (127.28.72.195). New cstate: current_term: 1 leader_uuid: "bd8bfee4d6aa442993aaef2327c34432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d96bf95c2245f6add7865c4f348e6a" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6235b352c0940a0bb459768fafb2e81" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 43071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd8bfee4d6aa442993aaef2327c34432" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35177 } health_report { overall_health: HEALTHY } } }
I20260923 17:26:35.376827 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.371433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.28.72.195:35177)
0923 17:26:35.371634 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:35.371638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:35.371655 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:26:35.371741 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:35.371744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:35.371879 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:35.372022 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:26:35.372410 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.373178 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.374549 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.374964 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.375056 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.375593 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:26:35.376227 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:26:35.376419 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:35.376582 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 17:26:35.376624 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:26:35.385550 14923 raft_consensus.cc:1261] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a [term 1 FOLLOWER]: Refusing update from remote peer bd8bfee4d6aa442993aaef2327c34432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:26:35.385587 15057 raft_consensus.cc:1261] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81 [term 1 FOLLOWER]: Refusing update from remote peer bd8bfee4d6aa442993aaef2327c34432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:26:35.386626 15265 consensus_queue.cc:1035] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6235b352c0940a0bb459768fafb2e81" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:26:35.387032 15263 consensus_queue.cc:1035] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54d96bf95c2245f6add7865c4f348e6a" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:26:35.418138 14779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:26:35.418422 14779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:26:35.422690 14779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc8a5ce4c083460682b8bceb7e0b7a59: Sending DeleteTablet for 3 replicas of tablet 2dc8c28c1aa64ae4857758e1a01be053
I20260923 17:26:35.424136 14903 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dc8c28c1aa64ae4857758e1a01be053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:35 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:57132
I20260923 17:26:35.424175 15171 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dc8c28c1aa64ae4857758e1a01be053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:35 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:46502
I20260923 17:26:35.425212 15278 tablet_replica.cc:330] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432: stopping tablet replica
I20260923 17:26:35.425200 15037 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dc8c28c1aa64ae4857758e1a01be053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:35 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:38728
I20260923 17:26:35.425817 15278 raft_consensus.cc:2229] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:26:35.426704 15278 raft_consensus.cc:2258] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:35.427295 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 14840
I20260923 17:26:35.429505 15278 ts_tablet_manager.cc:1905] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:35.440614 15278 ts_tablet_manager.cc:1918] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:35.441011 15278 log.cc:1196] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/2dc8c28c1aa64ae4857758e1a01be053
I20260923 17:26:35.441824 15278 ts_tablet_manager.cc:1939] T 2dc8c28c1aa64ae4857758e1a01be053 P bd8bfee4d6aa442993aaef2327c34432: Deleting consensus metadata
I20260923 17:26:35.444363 14765 catalog_manager.cc:4841] TS bd8bfee4d6aa442993aaef2327c34432 (127.28.72.195:35177): tablet 2dc8c28c1aa64ae4857758e1a01be053 (table test-table [id=066d02dd6c4040ae94caf3116b52bf56]) successfully deleted
I20260923 17:26:35.448479 15280 tablet_replica.cc:330] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81: stopping tablet replica
I20260923 17:26:35.449144 15280 raft_consensus.cc:2229] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:26:35.449927 15280 raft_consensus.cc:2258] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:35.451215 15279 tablet_replica.cc:330] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a: stopping tablet replica
I20260923 17:26:35.451725 15279 raft_consensus.cc:2229] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:26:35.452356 15279 raft_consensus.cc:2258] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:35.452507 15280 ts_tablet_manager.cc:1905] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:35.454463 15279 ts_tablet_manager.cc:1905] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 17:26:35 dist-test-slave-j0m4 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:26:35.460183 15280 ts_tablet_manager.cc:1918] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:35.460393 15280 log.cc:1196] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/2dc8c28c1aa64ae4857758e1a01be053
I20260923 17:26:35.460986 15280 ts_tablet_manager.cc:1939] T 2dc8c28c1aa64ae4857758e1a01be053 P d6235b352c0940a0bb459768fafb2e81: Deleting consensus metadata
I20260923 17:26:35.461813 15279 ts_tablet_manager.cc:1918] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:35.462036 15279 log.cc:1196] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/2dc8c28c1aa64ae4857758e1a01be053
I20260923 17:26:35.462641 15279 ts_tablet_manager.cc:1939] T 2dc8c28c1aa64ae4857758e1a01be053 P 54d96bf95c2245f6add7865c4f348e6a: Deleting consensus metadata
I20260923 17:26:35.463351 14765 catalog_manager.cc:4841] TS d6235b352c0940a0bb459768fafb2e81 (127.28.72.194:43071): tablet 2dc8c28c1aa64ae4857758e1a01be053 (table test-table [id=066d02dd6c4040ae94caf3116b52bf56]) successfully deleted
I20260923 17:26:35.464643 14763 catalog_manager.cc:4841] TS 54d96bf95c2245f6add7865c4f348e6a (127.28.72.193:46879): tablet 2dc8c28c1aa64ae4857758e1a01be053 (table test-table [id=066d02dd6c4040ae94caf3116b52bf56]) successfully deleted
I20260923 17:26:35.468366 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.430580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57148 (local address 127.28.72.193:46879)
0923 17:26:35.430761 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:35.430767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:35.447753 (+ 16986us) server_negotiation.cc:407] Connection header received
0923 17:26:35.447973 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:35.447982 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:35.448086 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:35.448209 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:35.450385 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.451726 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.453801 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.454484 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.454625 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.460784 (+ 6159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:35.460803 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:35.460836 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:35.460889 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:35.464733 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:35.465628 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:35.465642 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:35.465646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:35.465822 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:35.466366 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:35.466372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:35.466375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:35.466871 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:35.467232 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:35.467708 (+ 476us) server_negotiation.cc:299] Negotiation successful
0923 17:26:35.467976 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:26:35.469769 14963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:35.605832 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 14840
I20260923 17:26:35.624719 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 14974
Sep 23 17:26:35 dist-test-slave-j0m4 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:26:35.644116 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.625545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.28.72.194:43071)
0923 17:26:35.625767 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:35.625773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:35.625872 (+ 99us) server_negotiation.cc:407] Connection header received
0923 17:26:35.626121 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:35.626126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:35.626200 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:35.626344 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:35.628056 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.628926 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.630835 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.631313 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.631428 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.637140 (+ 5712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:35.637160 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:35.637177 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:35.637363 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:35.640544 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:35.641499 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:35.641508 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:35.641511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:35.641662 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:35.642116 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:35.642120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:35.642123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:35.642542 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:35.642771 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:35.643533 (+ 762us) server_negotiation.cc:299] Negotiation successful
0923 17:26:35.643799 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 17:26:35.644968 15097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:35.737296 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 14974
I20260923 17:26:35.755651 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 15108
Sep 23 17:26:35 dist-test-slave-j0m4 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:26:35.775909 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:35.756697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54188 (local address 127.28.72.195:35177)
0923 17:26:35.756886 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:35.756891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:35.756908 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:26:35.756963 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:35.756967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:35.757032 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:35.757117 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:35.758971 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.760067 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.762061 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:35.762510 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:35.762612 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:35.768698 (+ 6086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:35.768712 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:35.768729 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:35.768787 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:35.772157 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:35.773119 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:35.773132 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:35.773134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:35.773320 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:35.773799 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:35.773804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:35.773806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:35.774336 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:35.774624 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:35.775093 (+ 469us) server_negotiation.cc:299] Negotiation successful
0923 17:26:35.775486 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:26:35.776795 15231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:35.907703 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 15108
I20260923 17:26:35.928143 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 14747
I20260923 17:26:35.929312 14809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:36.048381 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 14747
2026-09-23T17:26:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4650 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T17:26:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:26:36Z Disabled control of system clock
I20260923 17:26:36.130115 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:37141
--webserver_interface=127.28.72.254
--webserver_port=0
--builtin_ntp_servers=127.28.72.212:36673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:37141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:26:36.492786 15289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:36.493108 15289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:36.493398 15289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:36.501303 15289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:26:36.501430 15289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:36.501492 15289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:36.501578 15289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:26:36.501653 15289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:26:36.511348 15289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:37141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:37141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:36.512813 15289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:36.514673 15289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:36.527664 15294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:36.527627 15297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:36.527709 15295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:36.528635 15289 server_base.cc:1034] running on GCE node
I20260923 17:26:36.529654 15289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:36.531404 15289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:36.532655 15289 hybrid_clock.cc:648] HybridClock initialized: now 1790184396532610 us; error 40 us; skew 500 ppm
I20260923 17:26:36.533041 15289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:26:36.541025 15289 webserver.cc:460] Webserver started at http://127.28.72.254:40965/ using document root <none> and password file <none>
I20260923 17:26:36.541854 15289 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:36.542008 15289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:36.542354 15289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:36.545619 15289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "a202667494394fe88479e41166cbcffa"
format_stamp: "Formatted at 2026-09-23 17:26:36 on dist-test-slave-j0m4"
server_key: "7c7080a0aff03e5a6b411e1ae1c451e3"
server_key_iv: "dd98a1a6d73b96f3b5ad0dc63f14691a"
server_key_version: "encryptionkey@0"
I20260923 17:26:36.546532 15289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "a202667494394fe88479e41166cbcffa"
format_stamp: "Formatted at 2026-09-23 17:26:36 on dist-test-slave-j0m4"
server_key: "7c7080a0aff03e5a6b411e1ae1c451e3"
server_key_iv: "dd98a1a6d73b96f3b5ad0dc63f14691a"
server_key_version: "encryptionkey@0"
I20260923 17:26:36.552172 15289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 17:26:36.556038 15303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:36.557688 15289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:26:36.557906 15289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "a202667494394fe88479e41166cbcffa"
format_stamp: "Formatted at 2026-09-23 17:26:36 on dist-test-slave-j0m4"
server_key: "7c7080a0aff03e5a6b411e1ae1c451e3"
server_key_iv: "dd98a1a6d73b96f3b5ad0dc63f14691a"
server_key_version: "encryptionkey@0"
I20260923 17:26:36.558147 15289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:36.580251 15289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:36.581207 15289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:36.581524 15289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:36.600828 15289 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:37141
I20260923 17:26:36.600847 15354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:37141 every 8 connection(s)
I20260923 17:26:36.602258 15289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:26:36.606231 15355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:36.607986 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 15289
I20260923 17:26:36.608176 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:26:36.608644 28963 external_mini_cluster.cc:1428] Setting key 565aaa8a85da1470416b3430cbee7bc9
I20260923 17:26:36.620393 15355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa: Bootstrap starting.
I20260923 17:26:36.622972 15358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:36.610601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.28.72.254:37141)
0923 17:26:36.611795 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:36.611808 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:36.611851 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:26:36.613607 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:36.613639 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:36.614232 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:36.614649 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:36.615294 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:36.616609 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:36.618089 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:36.618861 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:36.619079 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:36.620463 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:36.620473 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:36.620489 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:36.621116 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:36.621202 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:36.621226 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:36.621354 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:36.621595 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 17:26:36.621939 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":479,"threads_started":1}
I20260923 17:26:36.624401 15355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:36.625818 15355 log.cc:824] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:36.628768 15355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa: No bootstrap required, opened a new log
I20260923 17:26:36.636402 15355 raft_consensus.cc:348] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a202667494394fe88479e41166cbcffa" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37141 } }
I20260923 17:26:36.636842 15355 raft_consensus.cc:374] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:36.636948 15355 raft_consensus.cc:729] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a202667494394fe88479e41166cbcffa, State: Initialized, Role: FOLLOWER
I20260923 17:26:36.637626 15355 consensus_queue.cc:260] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a202667494394fe88479e41166cbcffa" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37141 } }
I20260923 17:26:36.637876 15355 raft_consensus.cc:388] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:26:36.638016 15355 raft_consensus.cc:482] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:26:36.638180 15355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:36.640564 15355 raft_consensus.cc:504] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a202667494394fe88479e41166cbcffa" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37141 } }
I20260923 17:26:36.641122 15355 leader_election.cc:302] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a202667494394fe88479e41166cbcffa; no voters:
I20260923 17:26:36.641731 15355 leader_election.cc:288] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:26:36.642032 15360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:36.643508 15360 raft_consensus.cc:686] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [term 1 LEADER]: Becoming Leader. State: Replica: a202667494394fe88479e41166cbcffa, State: Running, Role: LEADER
I20260923 17:26:36.644125 15355 sys_catalog.cc:564] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:26:36.644057 15360 consensus_queue.cc:237] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a202667494394fe88479e41166cbcffa" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37141 } }
I20260923 17:26:36.647495 15362 sys_catalog.cc:455] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader a202667494394fe88479e41166cbcffa. Latest consensus state: current_term: 1 leader_uuid: "a202667494394fe88479e41166cbcffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a202667494394fe88479e41166cbcffa" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37141 } } }
I20260923 17:26:36.648041 15362 sys_catalog.cc:458] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:36.647848 15361 sys_catalog.cc:455] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a202667494394fe88479e41166cbcffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a202667494394fe88479e41166cbcffa" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37141 } } }
I20260923 17:26:36.648177 15361 sys_catalog.cc:458] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:36.649408 15369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:26:36.655920 15369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:26:36.666097 15369 catalog_manager.cc:1347] Generated new cluster ID: 8b5bd3e64c6c40e59a694635469487f2
I20260923 17:26:36.666241 15369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:26:36.681074 15369 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:26:36.682555 15369 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:26:36.692808 15369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa: Generated new TSK 0
I20260923 17:26:36.694135 15369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:26:36.702162 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37141
--builtin_ntp_servers=127.28.72.212:36673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:26:37.055513 15379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:37.055819 15379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:37.055975 15379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:37.063902 15379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:37.064025 15379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:37.064185 15379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:26:37.073882 15379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:37.075699 15379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:37.078096 15379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:37.091393 15384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:37.092857 15385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:37.093075 15387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:37.093825 15379 server_base.cc:1034] running on GCE node
I20260923 17:26:37.094501 15379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:37.095592 15379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:37.096825 15379 hybrid_clock.cc:648] HybridClock initialized: now 1790184397096770 us; error 50 us; skew 500 ppm
I20260923 17:26:37.097179 15379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:26:37.099587 15379 webserver.cc:460] Webserver started at http://127.28.72.193:36377/ using document root <none> and password file <none>
I20260923 17:26:37.100327 15379 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:37.100458 15379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:37.100864 15379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:37.104104 15379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "81c76f35a9de41b4b78d08f8c624da08"
format_stamp: "Formatted at 2026-09-23 17:26:37 on dist-test-slave-j0m4"
server_key: "63c64e2cfacb2839ad7adc2651418505"
server_key_iv: "111900ef1d182f7789dcc0e6114fe84d"
server_key_version: "encryptionkey@0"
I20260923 17:26:37.104959 15379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "81c76f35a9de41b4b78d08f8c624da08"
format_stamp: "Formatted at 2026-09-23 17:26:37 on dist-test-slave-j0m4"
server_key: "63c64e2cfacb2839ad7adc2651418505"
server_key_iv: "111900ef1d182f7789dcc0e6114fe84d"
server_key_version: "encryptionkey@0"
I20260923 17:26:37.110602 15379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:26:37.114542 15393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:37.116031 15379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:26:37.116225 15379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "81c76f35a9de41b4b78d08f8c624da08"
format_stamp: "Formatted at 2026-09-23 17:26:37 on dist-test-slave-j0m4"
server_key: "63c64e2cfacb2839ad7adc2651418505"
server_key_iv: "111900ef1d182f7789dcc0e6114fe84d"
server_key_version: "encryptionkey@0"
I20260923 17:26:37.116457 15379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:37.140355 15379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:37.141299 15379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:37.141629 15379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:37.142692 15379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:37.144414 15379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:37.144536 15379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:37.144671 15379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:37.144757 15379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:37.179193 15379 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:36391
I20260923 17:26:37.179291 15505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:36391 every 8 connection(s)
I20260923 17:26:37.180557 15379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:26:37.191002 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 15379
I20260923 17:26:37.191231 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:26:37.191783 28963 external_mini_cluster.cc:1428] Setting key 49ec6406d0e102138750f60c7b6baf2f
I20260923 17:26:37.194365 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:37.183233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:50545 (local address 127.28.72.254:37141)
0923 17:26:37.183681 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:37.183686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:37.184834 (+ 1148us) server_negotiation.cc:407] Connection header received
0923 17:26:37.186700 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:37.186706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:37.186804 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:37.186924 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:37.188135 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:37.188879 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:37.190491 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:37.191199 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:37.191342 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:37.193272 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:37.193280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:37.193288 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:37.193315 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:37.193374 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:37.193395 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:37.193505 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:37.193837 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 17:26:37.193968 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":244,"threads_started":1}
I20260923 17:26:37.195461 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37141
--builtin_ntp_servers=127.28.72.212:36673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:26:37.195773 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:37.183785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37141 (local address 127.28.72.193:50545)
0923 17:26:37.184671 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:37.184712 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:37.186407 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:37.187246 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:37.187256 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:37.187359 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:37.187963 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:37.187979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:37.189046 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:37.189050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:37.190316 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:37.190328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:37.191550 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:37.191562 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:37.191874 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:37.192744 (+ 870us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:26:37.192748 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:37.192808 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:26:37.192933 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:37.193512 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:37.193526 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:37.193536 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 17:26:37.193820 (+ 284us) client_negotiation.cc:240] Negotiation successful
0923 17:26:37.194345 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":257,"threads_started":1}
I20260923 17:26:37.197669 15506 heartbeater.cc:347] Connected to a master server at 127.28.72.254:37141
I20260923 17:26:37.198037 15506 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:37.198892 15506 heartbeater.cc:511] Master 127.28.72.254:37141 requested a full tablet report, sending...
I20260923 17:26:37.201454 15320 ts_manager.cc:194] Registered new tserver with Master: 81c76f35a9de41b4b78d08f8c624da08 (127.28.72.193:36391)
I20260923 17:26:37.203181 15320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.72.193:50545
W20260923 17:26:37.553707 15510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:37.554040 15510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:37.554145 15510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:37.562755 15510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:37.562879 15510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:37.563000 15510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:26:37.574210 15510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:37.575587 15510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:37.577519 15510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:37.591701 15518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:37.591794 15515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:37.591737 15516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:37.592883 15510 server_base.cc:1034] running on GCE node
I20260923 17:26:37.593600 15510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:37.594667 15510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:37.595938 15510 hybrid_clock.cc:648] HybridClock initialized: now 1790184397595865 us; error 53 us; skew 500 ppm
I20260923 17:26:37.596347 15510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:26:37.599048 15510 webserver.cc:460] Webserver started at http://127.28.72.194:36943/ using document root <none> and password file <none>
I20260923 17:26:37.599866 15510 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:37.600025 15510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:37.600412 15510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:37.604465 15510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "5756cd58497a4099bc25d1f848e66bb6"
format_stamp: "Formatted at 2026-09-23 17:26:37 on dist-test-slave-j0m4"
server_key: "09949439279222e39c5efda1a545a631"
server_key_iv: "15686ad488c6b0707a7fa16fc4b91546"
server_key_version: "encryptionkey@0"
I20260923 17:26:37.605540 15510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "5756cd58497a4099bc25d1f848e66bb6"
format_stamp: "Formatted at 2026-09-23 17:26:37 on dist-test-slave-j0m4"
server_key: "09949439279222e39c5efda1a545a631"
server_key_iv: "15686ad488c6b0707a7fa16fc4b91546"
server_key_version: "encryptionkey@0"
I20260923 17:26:37.613183 15510 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260923 17:26:37.617714 15524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:37.619513 15510 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:26:37.619719 15510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "5756cd58497a4099bc25d1f848e66bb6"
format_stamp: "Formatted at 2026-09-23 17:26:37 on dist-test-slave-j0m4"
server_key: "09949439279222e39c5efda1a545a631"
server_key_iv: "15686ad488c6b0707a7fa16fc4b91546"
server_key_version: "encryptionkey@0"
I20260923 17:26:37.619962 15510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:37.635963 15510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:37.636947 15510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:37.637337 15510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:37.638482 15510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:37.640328 15510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:37.640465 15510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:37.640620 15510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:37.640715 15510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:37.679157 15510 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:41053
I20260923 17:26:37.679211 15636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:41053 every 8 connection(s)
I20260923 17:26:37.680552 15510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:26:37.683194 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 15510
I20260923 17:26:37.683481 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:26:37.684048 28963 external_mini_cluster.cc:1428] Setting key 23bebe130db808c9b674d78b8f6f8c1b
I20260923 17:26:37.686875 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37141
--builtin_ntp_servers=127.28.72.212:36673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:26:37.697065 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:37.683516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:47975 (local address 127.28.72.254:37141)
0923 17:26:37.683762 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:37.683779 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:37.685433 (+ 1654us) server_negotiation.cc:407] Connection header received
0923 17:26:37.687514 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:37.687521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:37.687616 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:37.687733 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:37.689045 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:37.690078 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:37.692946 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:37.693731 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:37.693881 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:37.696146 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:37.696152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:37.696156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:37.696178 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:37.696215 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:37.696228 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:37.696301 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:37.696625 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 17:26:37.696741 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 17:26:37.698233 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:37.684096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37141 (local address 127.28.72.194:47975)
0923 17:26:37.685261 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:37.685313 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:37.687221 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:37.688010 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:37.688020 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:37.688119 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:37.688785 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:37.688826 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:37.690236 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:37.690245 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:37.692690 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:37.692713 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:37.694284 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:37.694294 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:37.694680 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:37.695641 (+ 961us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:26:37.695646 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:37.695711 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:26:37.695850 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:37.696382 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:37.696392 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:37.696398 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:26:37.696590 (+ 192us) client_negotiation.cc:240] Negotiation successful
0923 17:26:37.697076 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":377,"threads_started":1}
I20260923 17:26:37.700081 15637 heartbeater.cc:347] Connected to a master server at 127.28.72.254:37141
I20260923 17:26:37.700542 15637 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:37.701479 15637 heartbeater.cc:511] Master 127.28.72.254:37141 requested a full tablet report, sending...
I20260923 17:26:37.703312 15320 ts_manager.cc:194] Registered new tserver with Master: 5756cd58497a4099bc25d1f848e66bb6 (127.28.72.194:41053)
I20260923 17:26:37.704245 15320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.72.194:47975
W20260923 17:26:38.038499 15640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:38.038827 15640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:38.038991 15640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:38.046634 15640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:38.046792 15640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:38.046967 15640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:26:38.056649 15640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:38.058044 15640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:38.060039 15640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:38.074086 15646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:38.074131 15645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:38.074152 15648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:38.075088 15640 server_base.cc:1034] running on GCE node
I20260923 17:26:38.075884 15640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:38.076963 15640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:38.078229 15640 hybrid_clock.cc:648] HybridClock initialized: now 1790184398078158 us; error 51 us; skew 500 ppm
I20260923 17:26:38.078697 15640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:26:38.082059 15640 webserver.cc:460] Webserver started at http://127.28.72.195:41561/ using document root <none> and password file <none>
I20260923 17:26:38.082821 15640 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:38.082978 15640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:38.083366 15640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:38.086346 15640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "76b6ee5bd3754a339175ca01f62f9c70"
format_stamp: "Formatted at 2026-09-23 17:26:38 on dist-test-slave-j0m4"
server_key: "327b3e1d35c75aa4a112d44ab8ad6916"
server_key_iv: "c4bda577362f0fcdcbc87c1111720a92"
server_key_version: "encryptionkey@0"
I20260923 17:26:38.087232 15640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "76b6ee5bd3754a339175ca01f62f9c70"
format_stamp: "Formatted at 2026-09-23 17:26:38 on dist-test-slave-j0m4"
server_key: "327b3e1d35c75aa4a112d44ab8ad6916"
server_key_iv: "c4bda577362f0fcdcbc87c1111720a92"
server_key_version: "encryptionkey@0"
I20260923 17:26:38.092718 15640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:26:38.096662 15654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:38.098196 15640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:26:38.098490 15640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "76b6ee5bd3754a339175ca01f62f9c70"
format_stamp: "Formatted at 2026-09-23 17:26:38 on dist-test-slave-j0m4"
server_key: "327b3e1d35c75aa4a112d44ab8ad6916"
server_key_iv: "c4bda577362f0fcdcbc87c1111720a92"
server_key_version: "encryptionkey@0"
I20260923 17:26:38.098729 15640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:38.124706 15640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:38.125676 15640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:38.126010 15640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:38.127032 15640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:38.128582 15640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:38.128703 15640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:38.128829 15640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:38.128914 15640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:38.162534 15640 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:39839
I20260923 17:26:38.162770 15766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:39839 every 8 connection(s)
I20260923 17:26:38.163964 15640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:26:38.171144 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 15640
I20260923 17:26:38.171363 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:26:38.171898 28963 external_mini_cluster.cc:1428] Setting key 185114371fed708e8b38fe609287433c
I20260923 17:26:38.178606 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.166731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:42513 (local address 127.28.72.254:37141)
0923 17:26:38.166947 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:38.166953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:38.168541 (+ 1588us) server_negotiation.cc:407] Connection header received
0923 17:26:38.170378 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:38.170383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:38.170473 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:38.170570 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:38.172435 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.173580 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.175745 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.176177 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.176319 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.177783 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:38.177788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:38.177792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:38.177813 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:38.177849 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:38.177862 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:38.177937 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:38.178246 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 17:26:38.178353 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:26:38.180310 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.167382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37141 (local address 127.28.72.195:42513)
0923 17:26:38.168374 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:38.168416 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:38.170086 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:38.170932 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:38.170946 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:38.171098 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:38.172154 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:38.172185 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.173789 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.173794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:38.175573 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:38.175586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.176419 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.176426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:38.176655 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.177419 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:26:38.177422 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:38.177464 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:26:38.177556 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:38.177989 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:38.177999 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:38.178006 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:26:38.178177 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 17:26:38.178695 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":292,"threads_started":1}
I20260923 17:26:38.181972 15767 heartbeater.cc:347] Connected to a master server at 127.28.72.254:37141
I20260923 17:26:38.182322 15767 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:38.183269 15767 heartbeater.cc:511] Master 127.28.72.254:37141 requested a full tablet report, sending...
I20260923 17:26:38.184893 15320 ts_manager.cc:194] Registered new tserver with Master: 76b6ee5bd3754a339175ca01f62f9c70 (127.28.72.195:39839)
I20260923 17:26:38.185734 15320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.72.195:42513
I20260923 17:26:38.188005 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:26:38.199575 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.192348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38736 (local address 127.28.72.254:37141)
0923 17:26:38.192569 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:38.192578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:38.193076 (+ 498us) server_negotiation.cc:407] Connection header received
0923 17:26:38.193218 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:38.193222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:38.193361 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:38.193462 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:38.194083 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.194930 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.196500 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.196931 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.197062 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.198512 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:38.198516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:38.198519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:38.198539 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:38.198572 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:38.198583 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:38.198714 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:38.199058 (+ 344us) server_negotiation.cc:299] Negotiation successful
0923 17:26:38.199166 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:26:38.206784 15506 heartbeater.cc:503] Master 127.28.72.254:37141 was elected leader, sending a full tablet report...
I20260923 17:26:38.204399 15319 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:26:38.211833 15319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:26:38.233529 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.224653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:36391 (local address 127.0.0.1:51598)
0923 17:26:38.225707 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:38.225743 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:38.225933 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:38.226781 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:38.226794 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:38.226866 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:38.227338 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:38.227353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.229161 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.229166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:38.230645 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:38.230657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.232416 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.232425 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:38.232644 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.232703 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 17:26:38.232819 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 17:26:38.233008 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":3,"spinlock_wait_cycles":19456,"thread_start_us":686,"threads_started":1}
I20260923 17:26:38.234436 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.225066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.28.72.193:36391)
0923 17:26:38.225718 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:38.225724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:38.225883 (+ 159us) server_negotiation.cc:407] Connection header received
0923 17:26:38.226212 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:38.226223 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:38.226480 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:38.227522 (+ 1042us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:38.227681 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.229017 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.230849 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.232190 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.233466 (+ 1276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.233763 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:38.233862 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 17:26:38.234036 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":367,"threads_started":1}
I20260923 17:26:38.237407 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.226544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.28.72.195:39839)
0923 17:26:38.227544 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:38.227552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:38.227590 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:26:38.227685 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:38.227695 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:38.227964 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:38.228876 (+ 912us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:38.229215 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.231288 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.234121 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.236362 (+ 2241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.236510 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.236793 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:38.236933 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 17:26:38.237091 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":614,"threads_started":1}
I20260923 17:26:38.237425 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.226158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:39839 (local address 127.0.0.1:40196)
0923 17:26:38.226726 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:38.226743 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:38.226884 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:38.228176 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:38.228180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:38.228206 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:38.228674 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:38.228684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.231491 (+ 2807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.231499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:38.233742 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:38.233756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.236577 (+ 2821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.236583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:38.236772 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.236808 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:26:38.236898 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 17:26:38.237034 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"spinlock_wait_cycles":12544,"thread_start_us":273,"threads_started":1}
I20260923 17:26:38.238626 15441 tablet_service.cc:1467] Processing CreateTablet for tablet b191f22f06604af190b11e4fc69496f6 (DEFAULT_TABLE table=test-table [id=4ebddac6caca44ff822ea15e93ca09ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:38.240252 15441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b191f22f06604af190b11e4fc69496f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:38.240286 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.227616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:41053 (local address 127.0.0.1:40612)
0923 17:26:38.228120 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:38.228137 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:38.228266 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:38.233313 (+ 5047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:38.233326 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:38.233351 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:38.234279 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:38.234289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.235935 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.235941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:38.237891 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:38.237907 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.239538 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.239545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:38.239729 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.239776 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 17:26:38.239853 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 17:26:38.239999 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":361,"threads_started":1}
I20260923 17:26:38.241614 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.227018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.28.72.194:41053)
0923 17:26:38.232757 (+ 5739us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:38.232766 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:38.232796 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:26:38.232896 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:38.232903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:38.233117 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:38.233326 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:38.234438 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.235766 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.238076 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.239229 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.240773 (+ 1544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.241004 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:38.241089 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 17:26:38.241255 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5598,"thread_start_us":5469,"threads_started":1}
I20260923 17:26:38.241693 15702 tablet_service.cc:1467] Processing CreateTablet for tablet b191f22f06604af190b11e4fc69496f6 (DEFAULT_TABLE table=test-table [id=4ebddac6caca44ff822ea15e93ca09ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:38.243099 15702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b191f22f06604af190b11e4fc69496f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:38.245365 15572 tablet_service.cc:1467] Processing CreateTablet for tablet b191f22f06604af190b11e4fc69496f6 (DEFAULT_TABLE table=test-table [id=4ebddac6caca44ff822ea15e93ca09ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:38.246780 15572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b191f22f06604af190b11e4fc69496f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:38.255148 15785 tablet_bootstrap.cc:492] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08: Bootstrap starting.
I20260923 17:26:38.257290 15786 tablet_bootstrap.cc:492] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70: Bootstrap starting.
I20260923 17:26:38.259102 15785 tablet_bootstrap.cc:654] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:38.260829 15786 tablet_bootstrap.cc:654] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:38.260835 15785 log.cc:824] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:38.262023 15787 tablet_bootstrap.cc:492] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6: Bootstrap starting.
I20260923 17:26:38.262089 15786 log.cc:824] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:38.265627 15785 tablet_bootstrap.cc:492] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08: No bootstrap required, opened a new log
I20260923 17:26:38.265893 15786 tablet_bootstrap.cc:492] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70: No bootstrap required, opened a new log
I20260923 17:26:38.265913 15785 ts_tablet_manager.cc:1397] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260923 17:26:38.266044 15787 tablet_bootstrap.cc:654] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:38.266171 15786 ts_tablet_manager.cc:1397] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260923 17:26:38.267387 15787 log.cc:824] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:38.270919 15787 tablet_bootstrap.cc:492] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6: No bootstrap required, opened a new log
I20260923 17:26:38.271251 15787 ts_tablet_manager.cc:1397] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260923 17:26:38.274657 15786 raft_consensus.cc:348] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c76f35a9de41b4b78d08f8c624da08" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36391 } } peers { permanent_uuid: "76b6ee5bd3754a339175ca01f62f9c70" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 39839 } } peers { permanent_uuid: "5756cd58497a4099bc25d1f848e66bb6" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 41053 } }
I20260923 17:26:38.275068 15786 raft_consensus.cc:374] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:38.275197 15786 raft_consensus.cc:729] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b6ee5bd3754a339175ca01f62f9c70, State: Initialized, Role: FOLLOWER
I20260923 17:26:38.275853 15786 consensus_queue.cc:260] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c76f35a9de41b4b78d08f8c624da08" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36391 } } peers { permanent_uuid: "76b6ee5bd3754a339175ca01f62f9c70" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 39839 } } peers { permanent_uuid: "5756cd58497a4099bc25d1f848e66bb6" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 41053 } }
I20260923 17:26:38.276374 15785 raft_consensus.cc:348] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c76f35a9de41b4b78d08f8c624da08" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36391 } } peers { permanent_uuid: "76b6ee5bd3754a339175ca01f62f9c70" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 39839 } } peers { permanent_uuid: "5756cd58497a4099bc25d1f848e66bb6" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 41053 } }
I20260923 17:26:38.276814 15785 raft_consensus.cc:374] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:38.276916 15785 raft_consensus.cc:729] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c76f35a9de41b4b78d08f8c624da08, State: Initialized, Role: FOLLOWER
I20260923 17:26:38.277067 15767 heartbeater.cc:503] Master 127.28.72.254:37141 was elected leader, sending a full tablet report...
I20260923 17:26:38.277493 15785 consensus_queue.cc:260] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c76f35a9de41b4b78d08f8c624da08" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36391 } } peers { permanent_uuid: "76b6ee5bd3754a339175ca01f62f9c70" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 39839 } } peers { permanent_uuid: "5756cd58497a4099bc25d1f848e66bb6" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 41053 } }
I20260923 17:26:38.277868 15786 ts_tablet_manager.cc:1428] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20260923 17:26:38.279434 15785 ts_tablet_manager.cc:1428] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260923 17:26:38.281765 15787 raft_consensus.cc:348] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c76f35a9de41b4b78d08f8c624da08" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36391 } } peers { permanent_uuid: "76b6ee5bd3754a339175ca01f62f9c70" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 39839 } } peers { permanent_uuid: "5756cd58497a4099bc25d1f848e66bb6" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 41053 } }
I20260923 17:26:38.282284 15787 raft_consensus.cc:374] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:38.282404 15787 raft_consensus.cc:729] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5756cd58497a4099bc25d1f848e66bb6, State: Initialized, Role: FOLLOWER
I20260923 17:26:38.283190 15787 consensus_queue.cc:260] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c76f35a9de41b4b78d08f8c624da08" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36391 } } peers { permanent_uuid: "76b6ee5bd3754a339175ca01f62f9c70" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 39839 } } peers { permanent_uuid: "5756cd58497a4099bc25d1f848e66bb6" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 41053 } }
I20260923 17:26:38.284559 15637 heartbeater.cc:503] Master 127.28.72.254:37141 was elected leader, sending a full tablet report...
I20260923 17:26:38.285158 15787 ts_tablet_manager.cc:1428] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260923 17:26:38.415884 15768 tablet.cc:2369] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:26:38.432729 15638 tablet.cc:2369] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:26:38.433032 15507 tablet.cc:2369] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:26:38.447937 15793 raft_consensus.cc:482] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:26:38.448299 15793 raft_consensus.cc:504] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c76f35a9de41b4b78d08f8c624da08" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36391 } } peers { permanent_uuid: "76b6ee5bd3754a339175ca01f62f9c70" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 39839 } } peers { permanent_uuid: "5756cd58497a4099bc25d1f848e66bb6" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 41053 } }
I20260923 17:26:38.450114 15793 leader_election.cc:288] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81c76f35a9de41b4b78d08f8c624da08 (127.28.72.193:36391), 76b6ee5bd3754a339175ca01f62f9c70 (127.28.72.195:39839)
I20260923 17:26:38.459527 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.451943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:54959 (local address 127.28.72.195:39839)
0923 17:26:38.452210 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:38.452217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:38.452652 (+ 435us) server_negotiation.cc:407] Connection header received
0923 17:26:38.452791 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:38.452797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:38.452875 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:38.452976 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:38.453519 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.454456 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.457051 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.458320 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.458492 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.458562 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:38.459075 (+ 513us) server_negotiation.cc:299] Negotiation successful
0923 17:26:38.459230 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 17:26:38.459499 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.450520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:57669 (local address 127.28.72.193:36391)
0923 17:26:38.450799 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:38.450806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:38.451499 (+ 693us) server_negotiation.cc:407] Connection header received
0923 17:26:38.451734 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:38.451738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:38.451811 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:38.451911 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:38.452684 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.454046 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.457051 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.458320 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.458493 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.458562 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:38.459075 (+ 513us) server_negotiation.cc:299] Negotiation successful
0923 17:26:38.459230 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260923 17:26:38.459669 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.450546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:36391 (local address 127.28.72.194:57669)
0923 17:26:38.451384 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:38.451402 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:38.451566 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:38.451939 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:38.451944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:38.451972 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:38.452465 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:38.452478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.454249 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.454253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:38.456765 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:38.456776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.458579 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.458585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:38.458840 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.458933 (+ 93us) client_negotiation.cc:769] Sending connection context
0923 17:26:38.459031 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 17:26:38.459289 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":156,"thread_start_us":415,"threads_started":1}
I20260923 17:26:38.460157 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.451616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:39839 (local address 127.28.72.194:54959)
0923 17:26:38.452541 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:38.452560 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:38.452672 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:38.453000 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:38.453003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:38.453021 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:38.453308 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:38.453315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.454607 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.454613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:38.456748 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:38.456766 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.458572 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.458579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:38.458842 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.458942 (+ 100us) client_negotiation.cc:769] Sending connection context
0923 17:26:38.459031 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 17:26:38.459289 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":165,"thread_start_us":567,"threads_started":1}
I20260923 17:26:38.460737 15461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b191f22f06604af190b11e4fc69496f6" candidate_uuid: "5756cd58497a4099bc25d1f848e66bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81c76f35a9de41b4b78d08f8c624da08" is_pre_election: true
I20260923 17:26:38.461301 15461 raft_consensus.cc:2454] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5756cd58497a4099bc25d1f848e66bb6 in term 0.
I20260923 17:26:38.461228 15722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b191f22f06604af190b11e4fc69496f6" candidate_uuid: "5756cd58497a4099bc25d1f848e66bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b6ee5bd3754a339175ca01f62f9c70" is_pre_election: true
I20260923 17:26:38.461669 15722 raft_consensus.cc:2454] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5756cd58497a4099bc25d1f848e66bb6 in term 0.
I20260923 17:26:38.461999 15527 leader_election.cc:302] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5756cd58497a4099bc25d1f848e66bb6, 81c76f35a9de41b4b78d08f8c624da08; no voters:
I20260923 17:26:38.462631 15793 raft_consensus.cc:2781] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:26:38.462805 15793 raft_consensus.cc:482] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:26:38.462918 15793 raft_consensus.cc:3037] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:38.465211 15793 raft_consensus.cc:504] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c76f35a9de41b4b78d08f8c624da08" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36391 } } peers { permanent_uuid: "76b6ee5bd3754a339175ca01f62f9c70" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 39839 } } peers { permanent_uuid: "5756cd58497a4099bc25d1f848e66bb6" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 41053 } }
I20260923 17:26:38.465781 15793 leader_election.cc:288] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [CANDIDATE]: Term 1 election: Requested vote from peers 81c76f35a9de41b4b78d08f8c624da08 (127.28.72.193:36391), 76b6ee5bd3754a339175ca01f62f9c70 (127.28.72.195:39839)
I20260923 17:26:38.466449 15722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b191f22f06604af190b11e4fc69496f6" candidate_uuid: "5756cd58497a4099bc25d1f848e66bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b6ee5bd3754a339175ca01f62f9c70"
I20260923 17:26:38.466467 15461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b191f22f06604af190b11e4fc69496f6" candidate_uuid: "5756cd58497a4099bc25d1f848e66bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81c76f35a9de41b4b78d08f8c624da08"
I20260923 17:26:38.466682 15722 raft_consensus.cc:3037] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:38.466681 15461 raft_consensus.cc:3037] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:38.470017 15461 raft_consensus.cc:2454] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5756cd58497a4099bc25d1f848e66bb6 in term 1.
I20260923 17:26:38.470028 15722 raft_consensus.cc:2454] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5756cd58497a4099bc25d1f848e66bb6 in term 1.
I20260923 17:26:38.470474 15527 leader_election.cc:302] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5756cd58497a4099bc25d1f848e66bb6, 81c76f35a9de41b4b78d08f8c624da08; no voters:
I20260923 17:26:38.470772 15793 raft_consensus.cc:2781] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:38.471282 15793 raft_consensus.cc:686] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [term 1 LEADER]: Becoming Leader. State: Replica: 5756cd58497a4099bc25d1f848e66bb6, State: Running, Role: LEADER
I20260923 17:26:38.471869 15793 consensus_queue.cc:237] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c76f35a9de41b4b78d08f8c624da08" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36391 } } peers { permanent_uuid: "76b6ee5bd3754a339175ca01f62f9c70" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 39839 } } peers { permanent_uuid: "5756cd58497a4099bc25d1f848e66bb6" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 41053 } }
I20260923 17:26:38.476752 15318 catalog_manager.cc:5495] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5756cd58497a4099bc25d1f848e66bb6 (127.28.72.194). New cstate: current_term: 1 leader_uuid: "5756cd58497a4099bc25d1f848e66bb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c76f35a9de41b4b78d08f8c624da08" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76b6ee5bd3754a339175ca01f62f9c70" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 39839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5756cd58497a4099bc25d1f848e66bb6" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 41053 } health_report { overall_health: HEALTHY } } }
I20260923 17:26:38.539453 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.534134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40622 (local address 127.28.72.194:41053)
0923 17:26:38.534337 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:38.534342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:38.534464 (+ 122us) server_negotiation.cc:407] Connection header received
0923 17:26:38.534629 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:38.534634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:38.534721 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:38.534853 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:26:38.535213 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.535915 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.537331 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.537745 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.537844 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.538326 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:26:38.538837 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:26:38.538981 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:38.539159 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 17:26:38.539206 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:26:38.548269 15461 raft_consensus.cc:1261] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08 [term 1 FOLLOWER]: Refusing update from remote peer 5756cd58497a4099bc25d1f848e66bb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:26:38.548305 15722 raft_consensus.cc:1261] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70 [term 1 FOLLOWER]: Refusing update from remote peer 5756cd58497a4099bc25d1f848e66bb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:26:38.549320 15796 consensus_queue.cc:1035] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b6ee5bd3754a339175ca01f62f9c70" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:26:38.549722 15793 consensus_queue.cc:1035] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81c76f35a9de41b4b78d08f8c624da08" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:26:38.562825 15799 mvcc.cc:205] Tried to move back new op lower bound from 7332595296442212352 to 7332595296145412096. Current Snapshot: MvccSnapshot[applied={T|T < 7332595296442212352 or (T in {7332595296442212352})}]
I20260923 17:26:38.563200 15800 mvcc.cc:205] Tried to move back new op lower bound from 7332595296442212352 to 7332595296145412096. Current Snapshot: MvccSnapshot[applied={T|T < 7332595296442212352 or (T in {7332595296442212352})}]
I20260923 17:26:38.581243 15318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:26:38.581576 15318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:26:38.585726 15318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a202667494394fe88479e41166cbcffa: Sending DeleteTablet for 3 replicas of tablet b191f22f06604af190b11e4fc69496f6
I20260923 17:26:38.587056 15441 tablet_service.cc:1514] Processing DeleteTablet for tablet b191f22f06604af190b11e4fc69496f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:38 UTC) from {username='slave'} at 127.0.0.1:51598
I20260923 17:26:38.587210 15572 tablet_service.cc:1514] Processing DeleteTablet for tablet b191f22f06604af190b11e4fc69496f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:38 UTC) from {username='slave'} at 127.0.0.1:40612
I20260923 17:26:38.587414 15702 tablet_service.cc:1514] Processing DeleteTablet for tablet b191f22f06604af190b11e4fc69496f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:38 UTC) from {username='slave'} at 127.0.0.1:40196
I20260923 17:26:38.589339 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 15379
I20260923 17:26:38.598769 15811 tablet_replica.cc:330] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70: stopping tablet replica
I20260923 17:26:38.599663 15811 raft_consensus.cc:2229] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:26:38.600574 15811 raft_consensus.cc:2258] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:38.601333 15809 tablet_replica.cc:330] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08: stopping tablet replica
I20260923 17:26:38.601612 15810 tablet_replica.cc:330] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6: stopping tablet replica
I20260923 17:26:38.602044 15810 raft_consensus.cc:2229] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:26:38.602403 15809 raft_consensus.cc:2229] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:26:38.602828 15810 raft_consensus.cc:2258] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:38.605067 15810 ts_tablet_manager.cc:1905] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:38.605062 15811 ts_tablet_manager.cc:1905] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:38.605683 15809 raft_consensus.cc:2258] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:38.609405 15809 ts_tablet_manager.cc:1905] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:38.614190 15810 ts_tablet_manager.cc:1918] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:38.614487 15810 log.cc:1196] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/b191f22f06604af190b11e4fc69496f6
I20260923 17:26:38.615176 15810 ts_tablet_manager.cc:1939] T b191f22f06604af190b11e4fc69496f6 P 5756cd58497a4099bc25d1f848e66bb6: Deleting consensus metadata
I20260923 17:26:38.617295 15306 catalog_manager.cc:4841] TS 5756cd58497a4099bc25d1f848e66bb6 (127.28.72.194:41053): tablet b191f22f06604af190b11e4fc69496f6 (table test-table [id=4ebddac6caca44ff822ea15e93ca09ef]) successfully deleted
I20260923 17:26:38.620185 15809 ts_tablet_manager.cc:1918] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:38.620419 15809 log.cc:1196] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/b191f22f06604af190b11e4fc69496f6
I20260923 17:26:38.620702 15811 ts_tablet_manager.cc:1918] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:38.620975 15811 log.cc:1196] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/b191f22f06604af190b11e4fc69496f6
I20260923 17:26:38.621177 15809 ts_tablet_manager.cc:1939] T b191f22f06604af190b11e4fc69496f6 P 81c76f35a9de41b4b78d08f8c624da08: Deleting consensus metadata
I20260923 17:26:38.621611 15811 ts_tablet_manager.cc:1939] T b191f22f06604af190b11e4fc69496f6 P 76b6ee5bd3754a339175ca01f62f9c70: Deleting consensus metadata
I20260923 17:26:38.623256 15306 catalog_manager.cc:4841] TS 81c76f35a9de41b4b78d08f8c624da08 (127.28.72.193:36391): tablet b191f22f06604af190b11e4fc69496f6 (table test-table [id=4ebddac6caca44ff822ea15e93ca09ef]) successfully deleted
I20260923 17:26:38.623606 15306 catalog_manager.cc:4841] TS 76b6ee5bd3754a339175ca01f62f9c70 (127.28.72.195:39839): tablet b191f22f06604af190b11e4fc69496f6 (table test-table [id=4ebddac6caca44ff822ea15e93ca09ef]) successfully deleted
I20260923 17:26:38.629428 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.590859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51610 (local address 127.28.72.193:36391)
0923 17:26:38.591073 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:38.591080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:38.619710 (+ 28630us) server_negotiation.cc:407] Connection header received
0923 17:26:38.620315 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:38.620338 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:38.620483 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:38.620573 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:38.621459 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.622757 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.625426 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.626147 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.626291 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.627582 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:38.627596 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:38.627620 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:38.628487 (+ 867us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:38.628553 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:38.628582 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:38.628706 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:38.628881 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 17:26:38.629011 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 17:26:38.630204 15501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:38.723199 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 15379
I20260923 17:26:38.748402 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 15510
I20260923 17:26:38.757020 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.749267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.28.72.194:41053)
0923 17:26:38.749461 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:38.749465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:38.749578 (+ 113us) server_negotiation.cc:407] Connection header received
0923 17:26:38.749791 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:38.749797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:38.749893 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:38.749988 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:38.750531 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.751475 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.753532 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.753982 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.754078 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.755536 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:38.755546 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:38.755560 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:38.756226 (+ 666us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:38.756272 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:38.756298 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:38.756386 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:38.756550 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 17:26:38.756636 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:26:38.757928 15632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:38.875232 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 15510
I20260923 17:26:38.894064 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 15640
I20260923 17:26:38.901309 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:38.894898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40204 (local address 127.28.72.195:39839)
0923 17:26:38.895148 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:38.895158 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:38.895182 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:26:38.895386 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:38.895390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:38.895492 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:38.895598 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:38.895986 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.896700 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.898020 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:38.898445 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:38.898553 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:38.899777 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:38.899786 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:38.899801 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:38.900476 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:38.900529 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:38.900556 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:38.900662 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:38.900806 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 17:26:38.900928 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 17:26:38.902113 15762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:38.993244 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 15640
I20260923 17:26:39.010428 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 15289
I20260923 17:26:39.011656 15350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:39.125241 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 15289
2026-09-23T17:26:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3076 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:26:39 dist-test-slave-j0m4 krb5kdc[15819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:26:39 dist-test-slave-j0m4 krb5kdc[15819](info): set up 2 sockets
Sep 23 17:26:39 dist-test-slave-j0m4 krb5kdc[15819](info): commencing operation
krb5kdc: starting...
W20260923 17:26:40.247709 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:26:40 dist-test-slave-j0m4 krb5kdc[15819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:26:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:26:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:26:40.516208 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:39237
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:40343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:39237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:40.870265 15832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:40.870601 15832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:40.870764 15832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:40.878326 15832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:26:40.878471 15832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:40.878561 15832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:40.878671 15832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:26:40.878746 15832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:26:40.889333 15832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:40343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:39237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:39237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:40.890786 15832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:40.892198 15832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:40.904668 15837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:40.904631 15840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:40.904716 15838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:40.905640 15832 server_base.cc:1034] running on GCE node
I20260923 17:26:40.906497 15832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:40.908087 15832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:40.909304 15832 hybrid_clock.cc:648] HybridClock initialized: now 1790184400909255 us; error 41 us; skew 500 ppm
I20260923 17:26:40.909698 15832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:40 dist-test-slave-j0m4 krb5kdc[15819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184400, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:40.913141 15832 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:26:40.915284 15832 webserver.cc:460] Webserver started at http://127.28.72.254:44933/ using document root <none> and password file <none>
I20260923 17:26:40.916100 15832 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:40.916322 15832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:40.916690 15832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:40.920142 15832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "ca5ffbe58df143b990f87b0174960bf9"
format_stamp: "Formatted at 2026-09-23 17:26:40 on dist-test-slave-j0m4"
server_key: "cb4396e774551b858282e1546256fd93"
server_key_iv: "3947297c23fa44f173305e1e54814c03"
server_key_version: "encryptionkey@0"
I20260923 17:26:40.921041 15832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "ca5ffbe58df143b990f87b0174960bf9"
format_stamp: "Formatted at 2026-09-23 17:26:40 on dist-test-slave-j0m4"
server_key: "cb4396e774551b858282e1546256fd93"
server_key_iv: "3947297c23fa44f173305e1e54814c03"
server_key_version: "encryptionkey@0"
I20260923 17:26:40.927002 15832 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 17:26:40.931092 15847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:40.932601 15832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:26:40.932808 15832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "ca5ffbe58df143b990f87b0174960bf9"
format_stamp: "Formatted at 2026-09-23 17:26:40 on dist-test-slave-j0m4"
server_key: "cb4396e774551b858282e1546256fd93"
server_key_iv: "3947297c23fa44f173305e1e54814c03"
server_key_version: "encryptionkey@0"
I20260923 17:26:40.933038 15832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:40.982913 15832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:40.988240 15832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:40.988596 15832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:41.009891 15832 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:39237
I20260923 17:26:41.009900 15898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:39237 every 8 connection(s)
I20260923 17:26:41.011629 15832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:26:41.013720 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 15832
I20260923 17:26:41.013935 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:26:41.014453 28963 external_mini_cluster.cc:1428] Setting key e169bccd5e7f31afa8a8cb7e487cd7b9
I20260923 17:26:41.016309 15899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:26:41 dist-test-slave-j0m4 krb5kdc[15819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:26:41.031811 15899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9: Bootstrap starting.
I20260923 17:26:41.036401 15899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:41.037745 15899 log.cc:824] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:41.041145 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:41.016915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.28.72.254:39237)
0923 17:26:41.017748 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:41.017760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:41.017804 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:26:41.019605 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:41.019629 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:41.020072 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:41.020455 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:41.022351 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:41.023899 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:41.025857 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:41.026695 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:41.026890 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:41.031881 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:41.031894 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:41.031919 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:41.031978 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:41.036082 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:41.037018 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:41.037027 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:41.037036 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:41.037257 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:41.037645 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:41.037650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:41.037653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:41.038427 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:41.038679 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:41.039282 (+ 603us) server_negotiation.cc:299] Negotiation successful
0923 17:26:41.039731 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":383,"threads_started":1}
I20260923 17:26:41.041543 15899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9: No bootstrap required, opened a new log
I20260923 17:26:41.050315 15899 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5ffbe58df143b990f87b0174960bf9" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 39237 } }
I20260923 17:26:41.050722 15899 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:41.050803 15899 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5ffbe58df143b990f87b0174960bf9, State: Initialized, Role: FOLLOWER
I20260923 17:26:41.051440 15899 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5ffbe58df143b990f87b0174960bf9" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 39237 } }
I20260923 17:26:41.051667 15899 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:26:41.051761 15899 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:26:41.051898 15899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:41.054344 15899 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5ffbe58df143b990f87b0174960bf9" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 39237 } }
I20260923 17:26:41.054849 15899 leader_election.cc:302] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca5ffbe58df143b990f87b0174960bf9; no voters:
I20260923 17:26:41.055454 15899 leader_election.cc:288] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:26:41.055616 15904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:41.056365 15904 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [term 1 LEADER]: Becoming Leader. State: Replica: ca5ffbe58df143b990f87b0174960bf9, State: Running, Role: LEADER
I20260923 17:26:41.056905 15904 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5ffbe58df143b990f87b0174960bf9" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 39237 } }
I20260923 17:26:41.057693 15899 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:26:41.060110 15906 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca5ffbe58df143b990f87b0174960bf9. Latest consensus state: current_term: 1 leader_uuid: "ca5ffbe58df143b990f87b0174960bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5ffbe58df143b990f87b0174960bf9" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 39237 } } }
I20260923 17:26:41.060207 15905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca5ffbe58df143b990f87b0174960bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5ffbe58df143b990f87b0174960bf9" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 39237 } } }
I20260923 17:26:41.060576 15906 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:41.060655 15905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:41.063467 15912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:26:41.068429 15912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:26:41.078650 15912 catalog_manager.cc:1347] Generated new cluster ID: 6562c3143990437191f72931c9b09e74
I20260923 17:26:41.078878 15912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:26:41.099619 15912 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:26:41.101563 15912 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:26:41.116308 15912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9: Generated new TSK 0
I20260923 17:26:41.117553 15912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.72.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
Entry for principal kudu/127.28.72.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab.
I20260923 17:26:41.187404 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:39237
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:40343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:41.546692 15925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:41.547405 15925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:41.547633 15925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:41.563715 15925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:41.563890 15925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:41.564157 15925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:26:41.580950 15925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:40343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.193.keytab
--principal=kudu/127.28.72.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:39237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:41.582847 15925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:41.584472 15925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:41.600199 15930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:41.600245 15931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:41.600273 15933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:41.601154 15925 server_base.cc:1034] running on GCE node
I20260923 17:26:41.601979 15925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:41.603085 15925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:41.604398 15925 hybrid_clock.cc:648] HybridClock initialized: now 1790184401604329 us; error 57 us; skew 500 ppm
I20260923 17:26:41.604755 15925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:41 dist-test-slave-j0m4 krb5kdc[15819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184401, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:41.608392 15925 init.cc:377] Logged in from keytab as kudu/127.28.72.193@KRBTEST.COM (short username kudu)
I20260923 17:26:41.610088 15925 webserver.cc:460] Webserver started at http://127.28.72.193:37115/ using document root <none> and password file <none>
I20260923 17:26:41.610908 15925 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:41.611063 15925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:41.611423 15925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:41.614717 15925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "cdd07cce382c40f8b3205551b43882ed"
format_stamp: "Formatted at 2026-09-23 17:26:41 on dist-test-slave-j0m4"
server_key: "9eaf3b4ba445c97392cacafed84ddca4"
server_key_iv: "40a54eaf887cba0c374877c7675ab77e"
server_key_version: "encryptionkey@0"
I20260923 17:26:41.615562 15925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "cdd07cce382c40f8b3205551b43882ed"
format_stamp: "Formatted at 2026-09-23 17:26:41 on dist-test-slave-j0m4"
server_key: "9eaf3b4ba445c97392cacafed84ddca4"
server_key_iv: "40a54eaf887cba0c374877c7675ab77e"
server_key_version: "encryptionkey@0"
I20260923 17:26:41.621423 15925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:26:41.625241 15940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:41.626700 15925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:26:41.626888 15925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "cdd07cce382c40f8b3205551b43882ed"
format_stamp: "Formatted at 2026-09-23 17:26:41 on dist-test-slave-j0m4"
server_key: "9eaf3b4ba445c97392cacafed84ddca4"
server_key_iv: "40a54eaf887cba0c374877c7675ab77e"
server_key_version: "encryptionkey@0"
I20260923 17:26:41.627179 15925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:41.646744 15925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:41.651494 15925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:41.651816 15925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:41.652882 15925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:41.654578 15925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:41.654691 15925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:41.654856 15925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:41.654934 15925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:41.692492 15925 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:36961
I20260923 17:26:41.692554 16052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:36961 every 8 connection(s)
I20260923 17:26:41.693969 15925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:26:41.695338 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 15925
I20260923 17:26:41.695539 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:26:41.696110 28963 external_mini_cluster.cc:1428] Setting key b48511618e6fe359b8e0e0d4f267f68e
Sep 23 17:26:41 dist-test-slave-j0m4 krb5kdc[15819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184401, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.193@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:26:41.721099 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:41.697286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:45997 (local address 127.28.72.254:39237)
0923 17:26:41.697792 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:41.697798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:41.699342 (+ 1544us) server_negotiation.cc:407] Connection header received
0923 17:26:41.702547 (+ 3205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:41.702552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:41.702629 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:41.702737 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:41.705222 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:41.706061 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:41.708862 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:41.709305 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:41.709580 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:41.714359 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:41.714366 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:41.714370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:41.714412 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:41.717448 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:41.718352 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:41.718358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:41.718360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:41.718495 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:41.719001 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:41.719005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:41.719007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:41.719367 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:41.719541 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:41.720385 (+ 844us) server_negotiation.cc:299] Negotiation successful
0923 17:26:41.720675 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":251,"threads_started":1}
I20260923 17:26:41.722250 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:41.698208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:39237 (local address 127.28.72.193:45997)
0923 17:26:41.699152 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:41.699207 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:41.702079 (+ 2872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:41.703054 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:41.703070 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:41.703999 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:41.704958 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:41.704991 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:41.706287 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:41.706295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:41.708649 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:41.708672 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:41.709609 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:41.709619 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:41.709933 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:41.710741 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:26:41.710779 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:41.714025 (+ 3246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:41.717636 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:41.717648 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:41.717670 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:41.718187 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:41.718623 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:41.718629 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:41.718633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:41.718859 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:41.719514 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:41.719524 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:41.719943 (+ 419us) client_negotiation.cc:769] Sending connection context
0923 17:26:41.720256 (+ 313us) client_negotiation.cc:240] Negotiation successful
0923 17:26:41.720913 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":328,"threads_started":1}
I20260923 17:26:41.724457 16053 heartbeater.cc:347] Connected to a master server at 127.28.72.254:39237
I20260923 17:26:41.724902 16053 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.72.194@KRBTEST.COM; defaulting to no policy
I20260923 17:26:41.725828 16053 heartbeater.cc:511] Master 127.28.72.254:39237 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.194@KRBTEST.COM" created.
I20260923 17:26:41.728753 15864 ts_manager.cc:194] Registered new tserver with Master: cdd07cce382c40f8b3205551b43882ed (127.28.72.193:36961)
I20260923 17:26:41.730378 15864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.193@KRBTEST.COM'} at 127.28.72.193:45997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
Entry for principal kudu/127.28.72.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab.
I20260923 17:26:41.763203 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:39237
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:40343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:42.120338 16059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:42.120671 16059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:42.120841 16059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:42.129187 16059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:42.129395 16059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:42.129575 16059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:26:42.140394 16059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:40343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.194.keytab
--principal=kudu/127.28.72.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:39237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:42.141826 16059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:42.143270 16059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:42.158178 16064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:42.158071 16065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:42.158109 16067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:42.158988 16059 server_base.cc:1034] running on GCE node
I20260923 17:26:42.159758 16059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:42.160820 16059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:42.162039 16059 hybrid_clock.cc:648] HybridClock initialized: now 1790184402161984 us; error 48 us; skew 500 ppm
I20260923 17:26:42.162396 16059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:42 dist-test-slave-j0m4 krb5kdc[15819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184402, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:42.166227 16059 init.cc:377] Logged in from keytab as kudu/127.28.72.194@KRBTEST.COM (short username kudu)
I20260923 17:26:42.168095 16059 webserver.cc:460] Webserver started at http://127.28.72.194:39739/ using document root <none> and password file <none>
I20260923 17:26:42.168916 16059 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:42.169060 16059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:42.169402 16059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:42.172698 16059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "ef6a7e689ffd4c8caec6ebf6cc480807"
format_stamp: "Formatted at 2026-09-23 17:26:42 on dist-test-slave-j0m4"
server_key: "41e0a8db5bbbb7076a26222bc8c38961"
server_key_iv: "d470f4c224b35c6b3eb4c08ce0c55f2b"
server_key_version: "encryptionkey@0"
I20260923 17:26:42.173535 16059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "ef6a7e689ffd4c8caec6ebf6cc480807"
format_stamp: "Formatted at 2026-09-23 17:26:42 on dist-test-slave-j0m4"
server_key: "41e0a8db5bbbb7076a26222bc8c38961"
server_key_iv: "d470f4c224b35c6b3eb4c08ce0c55f2b"
server_key_version: "encryptionkey@0"
I20260923 17:26:42.179412 16059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:26:42.183341 16074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:42.185092 16059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:26:42.185313 16059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "ef6a7e689ffd4c8caec6ebf6cc480807"
format_stamp: "Formatted at 2026-09-23 17:26:42 on dist-test-slave-j0m4"
server_key: "41e0a8db5bbbb7076a26222bc8c38961"
server_key_iv: "d470f4c224b35c6b3eb4c08ce0c55f2b"
server_key_version: "encryptionkey@0"
I20260923 17:26:42.185572 16059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:42.206629 16059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:42.211547 16059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:42.211867 16059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:42.213083 16059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:42.214689 16059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:42.214805 16059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:42.214960 16059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:42.215032 16059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:42.250214 16059 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:42873
I20260923 17:26:42.250270 16186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:42873 every 8 connection(s)
I20260923 17:26:42.251679 16059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:26:42.253753 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 16059
I20260923 17:26:42.253973 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:26:42.254503 28963 external_mini_cluster.cc:1428] Setting key 6bca82f171919d2d400c0801e2e9a34b
Sep 23 17:26:42 dist-test-slave-j0m4 krb5kdc[15819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184402, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.194@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:26:42.276182 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:42.254593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:58781 (local address 127.28.72.254:39237)
0923 17:26:42.255096 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:42.255102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:42.256373 (+ 1271us) server_negotiation.cc:407] Connection header received
0923 17:26:42.258417 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:42.258426 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:42.258522 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:42.258640 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:42.260542 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.261821 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.263815 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.264625 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.264782 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:42.269733 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:42.269739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:42.269742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:42.269783 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:42.272643 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:42.273584 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:42.273589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:42.273591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:42.273726 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:42.274208 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:42.274212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:42.274214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:42.274508 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:42.274673 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:42.275503 (+ 830us) server_negotiation.cc:299] Negotiation successful
0923 17:26:42.275764 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":240,"threads_started":1}
I20260923 17:26:42.277264 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:42.255272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:39237 (local address 127.28.72.194:58781)
0923 17:26:42.256158 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:42.256208 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:42.258119 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:42.258952 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:42.258970 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:42.259707 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:42.260337 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:42.260356 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.262026 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.262030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:42.263653 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:42.263669 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.264912 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.264923 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:42.265305 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:42.266157 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:26:42.266199 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:42.269402 (+ 3203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:42.272838 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:42.272850 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:42.272872 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:42.273421 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:42.273862 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:42.273867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:42.273870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:42.274077 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:42.274645 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:42.274654 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:42.275054 (+ 400us) client_negotiation.cc:769] Sending connection context
0923 17:26:42.275407 (+ 353us) client_negotiation.cc:240] Negotiation successful
0923 17:26:42.276009 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":358,"threads_started":1}
I20260923 17:26:42.279134 16187 heartbeater.cc:347] Connected to a master server at 127.28.72.254:39237
I20260923 17:26:42.279546 16187 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:42.280634 16187 heartbeater.cc:511] Master 127.28.72.254:39237 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.72.195@KRBTEST.COM; defaulting to no policy
I20260923 17:26:42.282752 15864 ts_manager.cc:194] Registered new tserver with Master: ef6a7e689ffd4c8caec6ebf6cc480807 (127.28.72.194:42873)
I20260923 17:26:42.283870 15864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.194@KRBTEST.COM'} at 127.28.72.194:58781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
Entry for principal kudu/127.28.72.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab.
I20260923 17:26:42.318548 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:39237
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:40343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:42.665223 16193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:42.665540 16193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:42.665637 16193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:42.673264 16193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:42.673517 16193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:42.673684 16193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:26:42.683965 16193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:40343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.195.keytab
--principal=kudu/127.28.72.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:39237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:42.685305 16193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:42.686642 16193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:42.702059 16198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:42.702250 16201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:42.702059 16199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:42.703011 16193 server_base.cc:1034] running on GCE node
I20260923 17:26:42.703743 16193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:42.704887 16193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:42.706115 16193 hybrid_clock.cc:648] HybridClock initialized: now 1790184402706066 us; error 47 us; skew 500 ppm
I20260923 17:26:42.706521 16193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:42 dist-test-slave-j0m4 krb5kdc[15819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184402, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:42.710009 16193 init.cc:377] Logged in from keytab as kudu/127.28.72.195@KRBTEST.COM (short username kudu)
I20260923 17:26:42.711661 16193 webserver.cc:460] Webserver started at http://127.28.72.195:33443/ using document root <none> and password file <none>
I20260923 17:26:42.712352 16193 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:42.712466 16193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:42.712728 16193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:42.716085 16193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "33745ce17b4e450d82c6bdce87a4456e"
format_stamp: "Formatted at 2026-09-23 17:26:42 on dist-test-slave-j0m4"
server_key: "7f0c1bb9bf6edf3d7782e16f47266066"
server_key_iv: "d52438176c5fbafe801c4cb9cecc1711"
server_key_version: "encryptionkey@0"
I20260923 17:26:42.716890 16193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "33745ce17b4e450d82c6bdce87a4456e"
format_stamp: "Formatted at 2026-09-23 17:26:42 on dist-test-slave-j0m4"
server_key: "7f0c1bb9bf6edf3d7782e16f47266066"
server_key_iv: "d52438176c5fbafe801c4cb9cecc1711"
server_key_version: "encryptionkey@0"
I20260923 17:26:42.722762 16193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 17:26:42.726676 16208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:42.728205 16193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:26:42.728399 16193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "33745ce17b4e450d82c6bdce87a4456e"
format_stamp: "Formatted at 2026-09-23 17:26:42 on dist-test-slave-j0m4"
server_key: "7f0c1bb9bf6edf3d7782e16f47266066"
server_key_iv: "d52438176c5fbafe801c4cb9cecc1711"
server_key_version: "encryptionkey@0"
I20260923 17:26:42.728659 16193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:42.734241 16053 heartbeater.cc:503] Master 127.28.72.254:39237 was elected leader, sending a full tablet report...
I20260923 17:26:42.763453 16193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:42.768178 16193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:42.768532 16193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:42.769559 16193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:42.771015 16193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:42.771092 16193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:42.771274 16193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:42.771329 16193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:42.805751 16193 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:35151
I20260923 17:26:42.805804 16320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:35151 every 8 connection(s)
I20260923 17:26:42.807256 16193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:26:42.815783 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 16193
I20260923 17:26:42.816061 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:26:42.816530 28963 external_mini_cluster.cc:1428] Setting key 552631939544f5175da8cb456d0c4a4c
Sep 23 17:26:42 dist-test-slave-j0m4 krb5kdc[15819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184402, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.195@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:26:42.828243 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:42.809863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:35451 (local address 127.28.72.254:39237)
0923 17:26:42.810377 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:42.810386 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:42.811996 (+ 1610us) server_negotiation.cc:407] Connection header received
0923 17:26:42.813883 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:42.813888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:42.813969 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:42.814074 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:42.815930 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.816839 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.818684 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.819280 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.819376 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:42.822810 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:42.822815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:42.822818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:42.822861 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:42.825377 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:42.826105 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:42.826109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:42.826111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:42.826238 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:42.826619 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:42.826623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:42.826625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:42.826900 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:42.827038 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:42.827759 (+ 721us) server_negotiation.cc:299] Negotiation successful
0923 17:26:42.827954 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":245,"threads_started":1}
I20260923 17:26:42.829233 16324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:42.810487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:39237 (local address 127.28.72.195:35451)
0923 17:26:42.811812 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:42.811861 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:42.813579 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:42.814313 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:42.814322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:42.815044 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:42.815730 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:42.815750 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.817006 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.817012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:42.818529 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:42.818544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.819511 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.819517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:42.819724 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:42.820391 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:26:42.820418 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:42.822586 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:42.825528 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:42.825535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:42.825551 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:42.825992 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:42.826334 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:42.826337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:42.826339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:42.826524 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:42.827018 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:42.827023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:42.827424 (+ 401us) client_negotiation.cc:769] Sending connection context
0923 17:26:42.827666 (+ 242us) client_negotiation.cc:240] Negotiation successful
0923 17:26:42.828076 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"thread_start_us":719,"threads_started":1}
I20260923 17:26:42.830809 16321 heartbeater.cc:347] Connected to a master server at 127.28.72.254:39237
I20260923 17:26:42.831202 16321 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:42.832052 16321 heartbeater.cc:511] Master 127.28.72.254:39237 requested a full tablet report, sending...
I20260923 17:26:42.833562 15864 ts_manager.cc:194] Registered new tserver with Master: 33745ce17b4e450d82c6bdce87a4456e (127.28.72.195:35151)
I20260923 17:26:42.834443 15864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.72.195@KRBTEST.COM'} at 127.28.72.195:35451
I20260923 17:26:42.845073 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:26:42.862408 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:42.849443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.28.72.254:39237)
0923 17:26:42.849621 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:42.849626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:42.850027 (+ 401us) server_negotiation.cc:407] Connection header received
0923 17:26:42.850171 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:42.850175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:42.850272 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:42.850364 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:42.851889 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.852628 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.853883 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.854307 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.854402 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:42.856783 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:42.856788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:42.856792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:42.856824 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:42.859698 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:42.860425 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:42.860429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:42.860431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:42.860562 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:42.860930 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:42.860934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:42.860936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:42.861297 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:42.861531 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:42.861905 (+ 374us) server_negotiation.cc:299] Negotiation successful
0923 17:26:42.862098 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:26:42.867178 15864 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:26:42.874378 15864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:26:42.904016 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:42.887597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.28.72.195:35151)
0923 17:26:42.888543 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:42.888554 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:42.891285 (+ 2731us) server_negotiation.cc:407] Connection header received
0923 17:26:42.891652 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:42.891663 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:42.892127 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:42.892315 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:42.893564 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.895651 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.900340 (+ 4689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.902023 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.902154 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:42.902477 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:42.903363 (+ 886us) server_negotiation.cc:299] Negotiation successful
0923 17:26:42.903603 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":587,"threads_started":1}
I20260923 17:26:42.905006 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:42.888028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:35151 (local address 127.0.0.1:34364)
0923 17:26:42.891151 (+ 3123us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:42.891171 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:42.891425 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:42.892740 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:42.892751 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:42.892791 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:42.893254 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:42.893299 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.896894 (+ 3595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.896901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:42.899954 (+ 3053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:42.899971 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.902695 (+ 2724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.902704 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:42.903010 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:42.903082 (+ 72us) client_negotiation.cc:769] Sending connection context
0923 17:26:42.904270 (+ 1188us) client_negotiation.cc:240] Negotiation successful
0923 17:26:42.904549 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2944,"thread_start_us":2773,"threads_started":1}
I20260923 17:26:42.912914 16256 tablet_service.cc:1467] Processing CreateTablet for tablet e462db14279e475d896f030a76db4598 (DEFAULT_TABLE table=test-table [id=cea8c589a6294a21b29b2d56424dd01c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:42.914686 16256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e462db14279e475d896f030a76db4598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:42.917089 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:42.887856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:42873 (local address 127.0.0.1:43636)
0923 17:26:42.889581 (+ 1725us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:42.889653 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:42.889913 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:42.909673 (+ 19760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:42.909678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:42.909706 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:42.910124 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:42.910141 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.911987 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.911996 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:42.914237 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:42.914253 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.916192 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.916198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:42.916364 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:42.916406 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 17:26:42.916504 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 17:26:42.916662 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"mutex_wait_us":62,"thread_start_us":938,"threads_started":1}
I20260923 17:26:42.918507 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:42.892591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:36961 (local address 127.0.0.1:48070)
0923 17:26:42.905288 (+ 12697us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:42.905314 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:42.905458 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:42.909087 (+ 3629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:42.909096 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:42.909136 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:42.909361 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:42.909376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.911471 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.911478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:42.914829 (+ 3351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:42.914854 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.917573 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.917581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:42.917798 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:42.917844 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 17:26:42.917945 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 17:26:42.918120 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12587,"mutex_wait_us":2,"thread_start_us":15182,"threads_started":1}
I20260923 17:26:42.919353 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:42.891702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.28.72.193:36961)
0923 17:26:42.908404 (+ 16702us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:42.908412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:42.908452 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:26:42.908557 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:42.908565 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:42.908840 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:42.909081 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:42.909644 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.911284 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.915040 (+ 3756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.916693 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.917161 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:42.917456 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:42.918752 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0923 17:26:42.918985 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16519,"thread_start_us":16832,"threads_started":1}
I20260923 17:26:42.919778 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:42.888354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.28.72.194:42873)
0923 17:26:42.907650 (+ 19296us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:42.907661 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:42.907702 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:26:42.907826 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:42.907835 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:42.909431 (+ 1596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:42.910361 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:42.910542 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.911821 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.914454 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:42.915980 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:42.916141 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:42.916350 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:42.919048 (+ 2698us) server_negotiation.cc:299] Negotiation successful
0923 17:26:42.919280 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19123,"thread_start_us":19010,"threads_started":1}
I20260923 17:26:42.923877 16122 tablet_service.cc:1467] Processing CreateTablet for tablet e462db14279e475d896f030a76db4598 (DEFAULT_TABLE table=test-table [id=cea8c589a6294a21b29b2d56424dd01c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:42.923882 15988 tablet_service.cc:1467] Processing CreateTablet for tablet e462db14279e475d896f030a76db4598 (DEFAULT_TABLE table=test-table [id=cea8c589a6294a21b29b2d56424dd01c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:42.925261 15988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e462db14279e475d896f030a76db4598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:42.925386 16122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e462db14279e475d896f030a76db4598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:42.932623 16340 tablet_bootstrap.cc:492] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e: Bootstrap starting.
I20260923 17:26:42.936913 16340 tablet_bootstrap.cc:654] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:42.938349 16340 log.cc:824] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:42.939052 16342 tablet_bootstrap.cc:492] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807: Bootstrap starting.
I20260923 17:26:42.942135 16342 tablet_bootstrap.cc:654] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:42.943727 16342 log.cc:824] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:42.943960 16341 tablet_bootstrap.cc:492] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed: Bootstrap starting.
I20260923 17:26:42.948467 16341 tablet_bootstrap.cc:654] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:42.949106 16340 tablet_bootstrap.cc:492] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e: No bootstrap required, opened a new log
I20260923 17:26:42.949400 16340 ts_tablet_manager.cc:1397] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260923 17:26:42.949797 16341 log.cc:824] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:42.949952 16342 tablet_bootstrap.cc:492] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807: No bootstrap required, opened a new log
I20260923 17:26:42.950212 16342 ts_tablet_manager.cc:1397] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260923 17:26:42.957993 16340 raft_consensus.cc:348] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33745ce17b4e450d82c6bdce87a4456e" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35151 } } peers { permanent_uuid: "cdd07cce382c40f8b3205551b43882ed" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36961 } } peers { permanent_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 42873 } }
I20260923 17:26:42.958397 16340 raft_consensus.cc:374] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:42.958494 16341 tablet_bootstrap.cc:492] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed: No bootstrap required, opened a new log
I20260923 17:26:42.958489 16340 raft_consensus.cc:729] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33745ce17b4e450d82c6bdce87a4456e, State: Initialized, Role: FOLLOWER
I20260923 17:26:42.958841 16341 ts_tablet_manager.cc:1397] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260923 17:26:42.959262 16340 consensus_queue.cc:260] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33745ce17b4e450d82c6bdce87a4456e" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35151 } } peers { permanent_uuid: "cdd07cce382c40f8b3205551b43882ed" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36961 } } peers { permanent_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 42873 } }
I20260923 17:26:42.960415 16321 heartbeater.cc:503] Master 127.28.72.254:39237 was elected leader, sending a full tablet report...
I20260923 17:26:42.961012 16340 ts_tablet_manager.cc:1428] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e: Time spent starting tablet: real 0.011s user 0.004s sys 0.009s
I20260923 17:26:42.961817 16342 raft_consensus.cc:348] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33745ce17b4e450d82c6bdce87a4456e" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35151 } } peers { permanent_uuid: "cdd07cce382c40f8b3205551b43882ed" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36961 } } peers { permanent_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 42873 } }
I20260923 17:26:42.962317 16342 raft_consensus.cc:374] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:42.962446 16342 raft_consensus.cc:729] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef6a7e689ffd4c8caec6ebf6cc480807, State: Initialized, Role: FOLLOWER
I20260923 17:26:42.963138 16342 consensus_queue.cc:260] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33745ce17b4e450d82c6bdce87a4456e" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35151 } } peers { permanent_uuid: "cdd07cce382c40f8b3205551b43882ed" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36961 } } peers { permanent_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 42873 } }
I20260923 17:26:42.964362 16187 heartbeater.cc:503] Master 127.28.72.254:39237 was elected leader, sending a full tablet report...
I20260923 17:26:42.965036 16342 ts_tablet_manager.cc:1428] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260923 17:26:42.972177 16341 raft_consensus.cc:348] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33745ce17b4e450d82c6bdce87a4456e" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35151 } } peers { permanent_uuid: "cdd07cce382c40f8b3205551b43882ed" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36961 } } peers { permanent_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 42873 } }
I20260923 17:26:42.972616 16341 raft_consensus.cc:374] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:42.972735 16341 raft_consensus.cc:729] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd07cce382c40f8b3205551b43882ed, State: Initialized, Role: FOLLOWER
I20260923 17:26:42.973444 16341 consensus_queue.cc:260] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33745ce17b4e450d82c6bdce87a4456e" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35151 } } peers { permanent_uuid: "cdd07cce382c40f8b3205551b43882ed" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36961 } } peers { permanent_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 42873 } }
I20260923 17:26:42.975180 16341 ts_tablet_manager.cc:1428] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed: Time spent starting tablet: real 0.016s user 0.004s sys 0.008s
W20260923 17:26:43.003804 16188 tablet.cc:2369] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:26:43.059002 16322 tablet.cc:2369] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:26:43.117664 16347 raft_consensus.cc:482] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:26:43.117957 16347 raft_consensus.cc:504] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33745ce17b4e450d82c6bdce87a4456e" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35151 } } peers { permanent_uuid: "cdd07cce382c40f8b3205551b43882ed" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36961 } } peers { permanent_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 42873 } }
I20260923 17:26:43.119819 16347 leader_election.cc:288] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33745ce17b4e450d82c6bdce87a4456e (127.28.72.195:35151), cdd07cce382c40f8b3205551b43882ed (127.28.72.193:36961)
I20260923 17:26:43.127416 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:43.120173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:48305 (local address 127.28.72.195:35151)
0923 17:26:43.120494 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:43.120503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:43.121077 (+ 574us) server_negotiation.cc:407] Connection header received
0923 17:26:43.121292 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:43.121299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:43.121443 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:43.121595 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:43.122317 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.123293 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.125356 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.126318 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.126452 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:43.126507 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:43.126970 (+ 463us) server_negotiation.cc:299] Negotiation successful
0923 17:26:43.127145 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:26:43.127534 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:43.120167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:35151 (local address 127.28.72.194:48305)
0923 17:26:43.120961 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:43.120982 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:43.121130 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:43.121643 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:43.121667 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:43.121696 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:43.122151 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:43.122165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.123428 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.123434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:43.125169 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:43.125186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.126562 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.126569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:43.126758 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:43.126801 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 17:26:43.126899 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 17:26:43.127077 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":4,"thread_start_us":453,"threads_started":1}
I20260923 17:26:43.127811 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:43.121265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:38057 (local address 127.28.72.193:36961)
0923 17:26:43.121467 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:43.121473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:43.121837 (+ 364us) server_negotiation.cc:407] Connection header received
0923 17:26:43.121956 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:43.121960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:43.122042 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:43.122145 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:43.122809 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.123758 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.125740 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.126751 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.126930 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:43.126998 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:43.127430 (+ 432us) server_negotiation.cc:299] Negotiation successful
0923 17:26:43.127588 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:26:43.128155 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:43.121097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:36961 (local address 127.28.72.194:38057)
0923 17:26:43.121725 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:43.121742 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:43.121851 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:43.122159 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:43.122162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:43.122179 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:43.122611 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:43.122625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.123908 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.123912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:43.125568 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:43.125583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.126992 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.126998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:43.127229 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:43.127267 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 17:26:43.127377 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 17:26:43.127537 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":41,"thread_start_us":313,"threads_started":1}
I20260923 17:26:43.128748 16276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e462db14279e475d896f030a76db4598" candidate_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33745ce17b4e450d82c6bdce87a4456e" is_pre_election: true
I20260923 17:26:43.129067 16008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e462db14279e475d896f030a76db4598" candidate_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd07cce382c40f8b3205551b43882ed" is_pre_election: true
I20260923 17:26:43.129271 16276 raft_consensus.cc:2454] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6a7e689ffd4c8caec6ebf6cc480807 in term 0.
I20260923 17:26:43.129563 16008 raft_consensus.cc:2454] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6a7e689ffd4c8caec6ebf6cc480807 in term 0.
I20260923 17:26:43.130023 16077 leader_election.cc:302] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33745ce17b4e450d82c6bdce87a4456e, ef6a7e689ffd4c8caec6ebf6cc480807; no voters:
I20260923 17:26:43.130483 16347 raft_consensus.cc:2781] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:26:43.130636 16347 raft_consensus.cc:482] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:26:43.130735 16347 raft_consensus.cc:3037] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:43.133200 16347 raft_consensus.cc:504] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33745ce17b4e450d82c6bdce87a4456e" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35151 } } peers { permanent_uuid: "cdd07cce382c40f8b3205551b43882ed" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36961 } } peers { permanent_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 42873 } }
I20260923 17:26:43.133844 16347 leader_election.cc:288] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [CANDIDATE]: Term 1 election: Requested vote from peers 33745ce17b4e450d82c6bdce87a4456e (127.28.72.195:35151), cdd07cce382c40f8b3205551b43882ed (127.28.72.193:36961)
I20260923 17:26:43.134423 16276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e462db14279e475d896f030a76db4598" candidate_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33745ce17b4e450d82c6bdce87a4456e"
I20260923 17:26:43.134636 16276 raft_consensus.cc:3037] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:43.134578 16008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e462db14279e475d896f030a76db4598" candidate_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd07cce382c40f8b3205551b43882ed"
I20260923 17:26:43.134860 16008 raft_consensus.cc:3037] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:43.137162 16276 raft_consensus.cc:2454] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6a7e689ffd4c8caec6ebf6cc480807 in term 1.
I20260923 17:26:43.137425 16008 raft_consensus.cc:2454] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6a7e689ffd4c8caec6ebf6cc480807 in term 1.
I20260923 17:26:43.137646 16077 leader_election.cc:302] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33745ce17b4e450d82c6bdce87a4456e, ef6a7e689ffd4c8caec6ebf6cc480807; no voters:
I20260923 17:26:43.138002 16347 raft_consensus.cc:2781] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:43.138511 16347 raft_consensus.cc:686] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [term 1 LEADER]: Becoming Leader. State: Replica: ef6a7e689ffd4c8caec6ebf6cc480807, State: Running, Role: LEADER
I20260923 17:26:43.139014 16347 consensus_queue.cc:237] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33745ce17b4e450d82c6bdce87a4456e" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35151 } } peers { permanent_uuid: "cdd07cce382c40f8b3205551b43882ed" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36961 } } peers { permanent_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 42873 } }
I20260923 17:26:43.144039 15864 catalog_manager.cc:5495] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef6a7e689ffd4c8caec6ebf6cc480807 (127.28.72.194). New cstate: current_term: 1 leader_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33745ce17b4e450d82c6bdce87a4456e" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd07cce382c40f8b3205551b43882ed" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6a7e689ffd4c8caec6ebf6cc480807" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 42873 } health_report { overall_health: HEALTHY } } }
I20260923 17:26:43.161336 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:43.156030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.28.72.194:42873)
0923 17:26:43.156202 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:43.156207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:43.156323 (+ 116us) server_negotiation.cc:407] Connection header received
0923 17:26:43.156514 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:43.156518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:43.156598 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:43.156724 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:26:43.157042 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.157782 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.159346 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.159748 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.159844 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:43.160323 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:26:43.160838 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:26:43.160955 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:43.161049 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 17:26:43.161116 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:26:43.169734 16008 raft_consensus.cc:1261] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed [term 1 FOLLOWER]: Refusing update from remote peer ef6a7e689ffd4c8caec6ebf6cc480807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:26:43.169735 16276 raft_consensus.cc:1261] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e [term 1 FOLLOWER]: Refusing update from remote peer ef6a7e689ffd4c8caec6ebf6cc480807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:26:43.170840 16351 consensus_queue.cc:1035] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33745ce17b4e450d82c6bdce87a4456e" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 35151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:26:43.171353 16347 consensus_queue.cc:1035] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd07cce382c40f8b3205551b43882ed" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:26:43.182333 16354 mvcc.cc:205] Tried to move back new op lower bound from 7332595315373096960 to 7332595315261562880. Current Snapshot: MvccSnapshot[applied={T|T < 7332595315373096960}]
I20260923 17:26:43.184947 16356 mvcc.cc:205] Tried to move back new op lower bound from 7332595315373096960 to 7332595315261562880. Current Snapshot: MvccSnapshot[applied={T|T < 7332595315373096960}]
I20260923 17:26:43.200516 15864 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:26:43.200858 15864 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:26:43.204957 15864 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca5ffbe58df143b990f87b0174960bf9: Sending DeleteTablet for 3 replicas of tablet e462db14279e475d896f030a76db4598
I20260923 17:26:43.206215 16256 tablet_service.cc:1514] Processing DeleteTablet for tablet e462db14279e475d896f030a76db4598 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:43 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:34364
I20260923 17:26:43.206225 16122 tablet_service.cc:1514] Processing DeleteTablet for tablet e462db14279e475d896f030a76db4598 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:43 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:43636
I20260923 17:26:43.207012 15988 tablet_service.cc:1514] Processing DeleteTablet for tablet e462db14279e475d896f030a76db4598 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:43 UTC) from {username='kudu', principal='kudu/127.28.72.254@KRBTEST.COM'} at 127.0.0.1:48070
I20260923 17:26:43.207731 16365 tablet_replica.cc:330] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807: stopping tablet replica
I20260923 17:26:43.207931 16364 tablet_replica.cc:330] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e: stopping tablet replica
I20260923 17:26:43.208442 16365 raft_consensus.cc:2229] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:26:43.208751 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 15925
I20260923 17:26:43.209071 16364 raft_consensus.cc:2229] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:26:43.209892 16365 raft_consensus.cc:2258] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:43.210485 16364 raft_consensus.cc:2258] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:43.213214 16365 ts_tablet_manager.cc:1905] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:43.217468 16364 ts_tablet_manager.cc:1905] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:43.221415 16365 ts_tablet_manager.cc:1918] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:43.221740 16365 log.cc:1196] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/e462db14279e475d896f030a76db4598
I20260923 17:26:43.222445 16365 ts_tablet_manager.cc:1939] T e462db14279e475d896f030a76db4598 P ef6a7e689ffd4c8caec6ebf6cc480807: Deleting consensus metadata
I20260923 17:26:43.224627 15849 catalog_manager.cc:4841] TS ef6a7e689ffd4c8caec6ebf6cc480807 (127.28.72.194:42873): tablet e462db14279e475d896f030a76db4598 (table test-table [id=cea8c589a6294a21b29b2d56424dd01c]) successfully deleted
I20260923 17:26:43.228628 16366 tablet_replica.cc:330] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed: stopping tablet replica
I20260923 17:26:43.229085 16366 raft_consensus.cc:2229] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:26:43.229650 16366 raft_consensus.cc:2258] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:43.231827 16364 ts_tablet_manager.cc:1918] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:43.232059 16364 log.cc:1196] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/e462db14279e475d896f030a76db4598
I20260923 17:26:43.232298 16366 ts_tablet_manager.cc:1905] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:43.232628 16364 ts_tablet_manager.cc:1939] T e462db14279e475d896f030a76db4598 P 33745ce17b4e450d82c6bdce87a4456e: Deleting consensus metadata
Sep 23 17:26:43 dist-test-slave-j0m4 krb5kdc[15819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.193@KRBTEST.COM
I20260923 17:26:43.234656 15850 catalog_manager.cc:4841] TS 33745ce17b4e450d82c6bdce87a4456e (127.28.72.195:35151): tablet e462db14279e475d896f030a76db4598 (table test-table [id=cea8c589a6294a21b29b2d56424dd01c]) successfully deleted
I20260923 17:26:43.239822 16366 ts_tablet_manager.cc:1918] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:43.239995 16366 log.cc:1196] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/e462db14279e475d896f030a76db4598
I20260923 17:26:43.240510 16366 ts_tablet_manager.cc:1939] T e462db14279e475d896f030a76db4598 P cdd07cce382c40f8b3205551b43882ed: Deleting consensus metadata
I20260923 17:26:43.242422 15850 catalog_manager.cc:4841] TS cdd07cce382c40f8b3205551b43882ed (127.28.72.193:36961): tablet e462db14279e475d896f030a76db4598 (table test-table [id=cea8c589a6294a21b29b2d56424dd01c]) successfully deleted
I20260923 17:26:43.243849 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:43.210601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.28.72.193:36961)
0923 17:26:43.210852 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:43.210859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:43.221245 (+ 10386us) server_negotiation.cc:407] Connection header received
0923 17:26:43.221882 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:43.221891 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:43.222020 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:43.222117 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:43.225107 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.226298 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.228582 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.229336 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.229477 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:43.235520 (+ 6043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:43.235536 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:43.235553 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:43.235596 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:43.239823 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:43.240901 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:43.240910 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:43.240914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:43.241095 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:43.241539 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:43.241545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:43.241549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:43.242044 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:43.242368 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:43.243240 (+ 872us) server_negotiation.cc:299] Negotiation successful
0923 17:26:43.243443 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:26:43.244588 16048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:43.339185 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 15925
I20260923 17:26:43.359627 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 16059
Sep 23 17:26:43 dist-test-slave-j0m4 krb5kdc[15819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.194@KRBTEST.COM
I20260923 17:26:43.378957 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:43.360605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.28.72.194:42873)
0923 17:26:43.360784 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:43.360790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:43.360813 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:26:43.361020 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:43.361024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:43.361103 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:43.361198 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:43.362948 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.364170 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.366121 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.366605 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.366706 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:43.372470 (+ 5764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:43.372492 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:43.372509 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:43.372561 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:43.375861 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:43.376685 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:43.376696 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:43.376700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:43.376862 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:43.377265 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:43.377272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:43.377277 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:43.377664 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:43.378029 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:43.378307 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 17:26:43.378544 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:26:43.379813 16182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:43.503242 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 16059
I20260923 17:26:43.525027 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 16193
Sep 23 17:26:43 dist-test-slave-j0m4 krb5kdc[15819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.195@KRBTEST.COM
I20260923 17:26:43.542865 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:43.525962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34366 (local address 127.28.72.195:35151)
0923 17:26:43.526184 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:43.526190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:43.526213 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:26:43.526392 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:43.526399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:43.526490 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:43.526624 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:43.528188 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.528999 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.530576 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:43.531073 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:43.531404 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:43.536657 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:43.536681 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:43.536709 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:43.536766 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:43.539741 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:43.540569 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:43.540579 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:43.540581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:43.540748 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:43.541177 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:43.541183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:43.541186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:43.541580 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:43.541910 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:43.542245 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 17:26:43.542482 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:26:43.543591 16316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:43.636529 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 16193
I20260923 17:26:43.655575 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 15832
I20260923 17:26:43.656759 15894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:43.769409 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 15832
2026-09-23T17:26:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4645 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T17:26:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:26:43Z Disabled control of system clock
I20260923 17:26:43.852573 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:37091
--webserver_interface=127.28.72.254
--webserver_port=0
--builtin_ntp_servers=127.28.72.212:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:37091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:26:44.208750 16375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:44.209091 16375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:44.209179 16375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:44.216722 16375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:26:44.216872 16375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:44.216912 16375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:44.216964 16375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:26:44.217010 16375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:26:44.227064 16375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:37091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:37091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:44.228488 16375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:44.230288 16375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:44.242801 16381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:44.243048 16380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:44.242767 16383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:44.243637 16375 server_base.cc:1034] running on GCE node
I20260923 17:26:44.244617 16375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:44.246161 16375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:44.247463 16375 hybrid_clock.cc:648] HybridClock initialized: now 1790184404247366 us; error 40 us; skew 500 ppm
I20260923 17:26:44.247857 16375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:26:44.250316 16375 webserver.cc:460] Webserver started at http://127.28.72.254:34465/ using document root <none> and password file <none>
I20260923 17:26:44.251071 16375 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:44.251277 16375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:44.251633 16375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:44.254719 16375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "33adf2ff87d7453481fb9c5bbb95d4e5"
format_stamp: "Formatted at 2026-09-23 17:26:44 on dist-test-slave-j0m4"
server_key: "a0e7fe29b07e2c8b0b31ccabb8ae6c79"
server_key_iv: "610b8dbe2e311129cc8731dc852f89ba"
server_key_version: "encryptionkey@0"
I20260923 17:26:44.255620 16375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "33adf2ff87d7453481fb9c5bbb95d4e5"
format_stamp: "Formatted at 2026-09-23 17:26:44 on dist-test-slave-j0m4"
server_key: "a0e7fe29b07e2c8b0b31ccabb8ae6c79"
server_key_iv: "610b8dbe2e311129cc8731dc852f89ba"
server_key_version: "encryptionkey@0"
I20260923 17:26:44.261310 16375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:26:44.265482 16389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:44.267205 16375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:26:44.267393 16375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "33adf2ff87d7453481fb9c5bbb95d4e5"
format_stamp: "Formatted at 2026-09-23 17:26:44 on dist-test-slave-j0m4"
server_key: "a0e7fe29b07e2c8b0b31ccabb8ae6c79"
server_key_iv: "610b8dbe2e311129cc8731dc852f89ba"
server_key_version: "encryptionkey@0"
I20260923 17:26:44.267690 16375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:44.282691 16375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:44.283743 16375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:44.284083 16375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:44.304657 16375 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:37091
I20260923 17:26:44.304663 16440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:37091 every 8 connection(s)
I20260923 17:26:44.306176 16375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:26:44.309734 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 16375
I20260923 17:26:44.309926 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:26:44.310519 28963 external_mini_cluster.cc:1428] Setting key 8acdd4039a5406a1211be68192844653
I20260923 17:26:44.310838 16441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:44.325027 16441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5: Bootstrap starting.
I20260923 17:26:44.325857 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:44.312580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.28.72.254:37091)
0923 17:26:44.313795 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:44.313807 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:44.313853 (+ 46us) server_negotiation.cc:407] Connection header received
0923 17:26:44.316011 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:44.316039 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:44.316572 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:44.317024 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:44.317697 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:44.319311 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:44.320804 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:44.321547 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:44.321801 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:44.323083 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:44.323092 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:44.323159 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:44.323833 (+ 674us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:44.323929 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:44.323953 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:44.324095 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:44.324323 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 17:26:44.324675 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":429,"threads_started":1}
I20260923 17:26:44.328931 16441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:44.330009 16441 log.cc:824] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:44.333060 16441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5: No bootstrap required, opened a new log
I20260923 17:26:44.340847 16441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33adf2ff87d7453481fb9c5bbb95d4e5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37091 } }
I20260923 17:26:44.341287 16441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:44.341406 16441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33adf2ff87d7453481fb9c5bbb95d4e5, State: Initialized, Role: FOLLOWER
I20260923 17:26:44.342135 16441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33adf2ff87d7453481fb9c5bbb95d4e5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37091 } }
I20260923 17:26:44.342403 16441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:26:44.342553 16441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:26:44.342731 16441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:44.345108 16441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33adf2ff87d7453481fb9c5bbb95d4e5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37091 } }
I20260923 17:26:44.345672 16441 leader_election.cc:302] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33adf2ff87d7453481fb9c5bbb95d4e5; no voters:
I20260923 17:26:44.346308 16441 leader_election.cc:288] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:26:44.346483 16446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:44.347867 16446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [term 1 LEADER]: Becoming Leader. State: Replica: 33adf2ff87d7453481fb9c5bbb95d4e5, State: Running, Role: LEADER
I20260923 17:26:44.348489 16446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33adf2ff87d7453481fb9c5bbb95d4e5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37091 } }
I20260923 17:26:44.348779 16441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:26:44.351488 16447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33adf2ff87d7453481fb9c5bbb95d4e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33adf2ff87d7453481fb9c5bbb95d4e5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37091 } } }
I20260923 17:26:44.351466 16448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33adf2ff87d7453481fb9c5bbb95d4e5. Latest consensus state: current_term: 1 leader_uuid: "33adf2ff87d7453481fb9c5bbb95d4e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33adf2ff87d7453481fb9c5bbb95d4e5" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37091 } } }
I20260923 17:26:44.352001 16447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:44.352000 16448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:44.353049 16453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:26:44.358971 16453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:26:44.370046 16453 catalog_manager.cc:1347] Generated new cluster ID: d2fdf1205c0f4d318df44382d7fa72e1
I20260923 17:26:44.370250 16453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:26:44.378958 16453 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:26:44.380409 16453 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:26:44.391551 16453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5: Generated new TSK 0
I20260923 17:26:44.392796 16453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:26:44.404816 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.193:0
--local_ip_for_outbound_sockets=127.28.72.193
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37091
--builtin_ntp_servers=127.28.72.212:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:26:44.761734 16465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:44.762097 16465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:44.762259 16465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:44.771030 16465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:44.771229 16465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:44.771474 16465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.193
I20260923 17:26:44.781232 16465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.72.193
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.193
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:44.782773 16465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:44.784688 16465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:44.799073 16470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:44.799141 16471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:44.799633 16473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:44.800637 16465 server_base.cc:1034] running on GCE node
I20260923 17:26:44.801378 16465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:44.802439 16465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:44.803690 16465 hybrid_clock.cc:648] HybridClock initialized: now 1790184404803630 us; error 52 us; skew 500 ppm
I20260923 17:26:44.804057 16465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:26:44.806519 16465 webserver.cc:460] Webserver started at http://127.28.72.193:34977/ using document root <none> and password file <none>
I20260923 17:26:44.807646 16465 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:44.807801 16465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:44.808132 16465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:44.811345 16465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/data/instance:
uuid: "79100060896c4e2cbe665cdb2399e540"
format_stamp: "Formatted at 2026-09-23 17:26:44 on dist-test-slave-j0m4"
server_key: "0690c0d36463c743633b0979b3f2770b"
server_key_iv: "6d895a1044023713d9c77161fea2bdec"
server_key_version: "encryptionkey@0"
I20260923 17:26:44.812215 16465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance:
uuid: "79100060896c4e2cbe665cdb2399e540"
format_stamp: "Formatted at 2026-09-23 17:26:44 on dist-test-slave-j0m4"
server_key: "0690c0d36463c743633b0979b3f2770b"
server_key_iv: "6d895a1044023713d9c77161fea2bdec"
server_key_version: "encryptionkey@0"
I20260923 17:26:44.818022 16465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:26:44.822177 16479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:44.823832 16465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:26:44.824038 16465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/wal
uuid: "79100060896c4e2cbe665cdb2399e540"
format_stamp: "Formatted at 2026-09-23 17:26:44 on dist-test-slave-j0m4"
server_key: "0690c0d36463c743633b0979b3f2770b"
server_key_iv: "6d895a1044023713d9c77161fea2bdec"
server_key_version: "encryptionkey@0"
I20260923 17:26:44.824285 16465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:44.841164 16465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:44.842085 16465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:44.842417 16465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:44.843572 16465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:44.845111 16465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:44.845247 16465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:44.845374 16465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:44.845467 16465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:44.881524 16465 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.193:34441
I20260923 17:26:44.881620 16591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.193:34441 every 8 connection(s)
I20260923 17:26:44.882956 16465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/data/info.pb
I20260923 17:26:44.892380 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 16465
I20260923 17:26:44.892580 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/wal/instance
I20260923 17:26:44.893044 28963 external_mini_cluster.cc:1428] Setting key 2cbaeaf94e49ed694911235399d85d21
I20260923 17:26:44.896487 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.194:0
--local_ip_for_outbound_sockets=127.28.72.194
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37091
--builtin_ntp_servers=127.28.72.212:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:26:44.897308 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:44.885787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:46063 (local address 127.28.72.254:37091)
0923 17:26:44.886249 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:44.886254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:44.887661 (+ 1407us) server_negotiation.cc:407] Connection header received
0923 17:26:44.889605 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:44.889610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:44.889720 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:44.889842 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:44.891145 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:44.891957 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:44.893811 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:44.894419 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:44.894712 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:44.896466 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:44.896471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:44.896474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:44.896493 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:44.896533 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:44.896544 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:44.896678 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:44.896940 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 17:26:44.897047 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":238,"threads_started":1}
I20260923 17:26:44.898983 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:44.886511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37091 (local address 127.28.72.193:46063)
0923 17:26:44.887498 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:44.887534 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:44.889247 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:44.890105 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:44.890114 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:44.890211 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:44.890917 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:44.890936 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:44.892128 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:44.892132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:44.893618 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:44.893637 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:44.894778 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:44.894786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:44.895175 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:44.896044 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:26:44.896050 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:44.896116 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:26:44.896243 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:44.896695 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:44.896711 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:44.896723 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 17:26:44.896921 (+ 198us) client_negotiation.cc:240] Negotiation successful
0923 17:26:44.897424 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":263,"threads_started":1}
I20260923 17:26:44.904633 16592 heartbeater.cc:347] Connected to a master server at 127.28.72.254:37091
I20260923 17:26:44.905004 16592 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:44.905874 16592 heartbeater.cc:511] Master 127.28.72.254:37091 requested a full tablet report, sending...
I20260923 17:26:44.908242 16406 ts_manager.cc:194] Registered new tserver with Master: 79100060896c4e2cbe665cdb2399e540 (127.28.72.193:34441)
I20260923 17:26:44.909770 16406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.72.193:46063
W20260923 17:26:45.244387 16596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:45.244697 16596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:45.244845 16596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:45.252570 16596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:45.252684 16596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:45.252859 16596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.194
I20260923 17:26:45.262481 16596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.72.194
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.194
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:45.263881 16596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:45.265725 16596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:45.279592 16601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:45.279657 16602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:45.279753 16604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:45.280314 16596 server_base.cc:1034] running on GCE node
I20260923 17:26:45.280982 16596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:45.282001 16596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:45.283221 16596 hybrid_clock.cc:648] HybridClock initialized: now 1790184405283152 us; error 61 us; skew 500 ppm
I20260923 17:26:45.283557 16596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:26:45.285916 16596 webserver.cc:460] Webserver started at http://127.28.72.194:41655/ using document root <none> and password file <none>
I20260923 17:26:45.286689 16596 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:45.286829 16596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:45.287214 16596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:45.290233 16596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/data/instance:
uuid: "f4cbe347a56544abb2104ad935e56164"
format_stamp: "Formatted at 2026-09-23 17:26:45 on dist-test-slave-j0m4"
server_key: "1ffc25f4776ba28b359c505885190715"
server_key_iv: "c7e45399951d216f339d182741f5b89d"
server_key_version: "encryptionkey@0"
I20260923 17:26:45.291090 16596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance:
uuid: "f4cbe347a56544abb2104ad935e56164"
format_stamp: "Formatted at 2026-09-23 17:26:45 on dist-test-slave-j0m4"
server_key: "1ffc25f4776ba28b359c505885190715"
server_key_iv: "c7e45399951d216f339d182741f5b89d"
server_key_version: "encryptionkey@0"
I20260923 17:26:45.296788 16596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 17:26:45.300626 16610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:45.302057 16596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:26:45.302263 16596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/wal
uuid: "f4cbe347a56544abb2104ad935e56164"
format_stamp: "Formatted at 2026-09-23 17:26:45 on dist-test-slave-j0m4"
server_key: "1ffc25f4776ba28b359c505885190715"
server_key_iv: "c7e45399951d216f339d182741f5b89d"
server_key_version: "encryptionkey@0"
I20260923 17:26:45.302512 16596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:45.332739 16596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:45.333657 16596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:45.333977 16596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:45.335058 16596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:45.336608 16596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:45.336732 16596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:45.336876 16596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:45.336966 16596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:45.372089 16596 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.194:36331
I20260923 17:26:45.372143 16722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.194:36331 every 8 connection(s)
I20260923 17:26:45.373526 16596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/data/info.pb
I20260923 17:26:45.374979 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 16596
I20260923 17:26:45.375196 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/wal/instance
I20260923 17:26:45.375733 28963 external_mini_cluster.cc:1428] Setting key 35d60fde5d4188a11fb67a72af332d3f
I20260923 17:26:45.378937 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.72.195:0
--local_ip_for_outbound_sockets=127.28.72.195
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37091
--builtin_ntp_servers=127.28.72.212:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:26:45.387224 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:45.376650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.194:33523 (local address 127.28.72.254:37091)
0923 17:26:45.376886 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:45.376893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:45.378416 (+ 1523us) server_negotiation.cc:407] Connection header received
0923 17:26:45.380392 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:45.380398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:45.380500 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:45.380613 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:45.381745 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.382486 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.384150 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.384567 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.384699 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:45.386172 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:45.386177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:45.386180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:45.386199 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:45.386257 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:45.386276 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:45.386370 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:45.386723 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 17:26:45.386848 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:26:45.388624 16725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:45.377158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37091 (local address 127.28.72.194:33523)
0923 17:26:45.378260 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:45.378303 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:45.380116 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:45.380861 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:45.380869 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:45.380966 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:45.381557 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:45.381578 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.382639 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.382646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:45.383999 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:45.384012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.384808 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.384816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:45.385092 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:45.385826 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:26:45.385829 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:45.385876 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:26:45.385961 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:45.386427 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:45.386440 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:45.386450 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 17:26:45.386659 (+ 209us) client_negotiation.cc:240] Negotiation successful
0923 17:26:45.387152 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":401,"threads_started":1}
I20260923 17:26:45.390187 16723 heartbeater.cc:347] Connected to a master server at 127.28.72.254:37091
I20260923 17:26:45.390520 16723 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:45.391515 16723 heartbeater.cc:511] Master 127.28.72.254:37091 requested a full tablet report, sending...
I20260923 17:26:45.393154 16406 ts_manager.cc:194] Registered new tserver with Master: f4cbe347a56544abb2104ad935e56164 (127.28.72.194:36331)
I20260923 17:26:45.394101 16406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.72.194:33523
W20260923 17:26:45.728888 16726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:45.729182 16726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:45.729341 16726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:45.736656 16726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:45.736765 16726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:45.736934 16726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.72.195
I20260923 17:26:45.746053 16726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.72.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.72.195
--webserver_port=0
--tserver_master_addrs=127.28.72.254:37091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.72.195
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:45.747460 16726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:45.749414 16726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:45.763068 16734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:45.763219 16731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:45.763098 16732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:45.763679 16726 server_base.cc:1034] running on GCE node
I20260923 17:26:45.764367 16726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:45.765304 16726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:45.766523 16726 hybrid_clock.cc:648] HybridClock initialized: now 1790184405766453 us; error 65 us; skew 500 ppm
I20260923 17:26:45.766922 16726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:26:45.769517 16726 webserver.cc:460] Webserver started at http://127.28.72.195:46299/ using document root <none> and password file <none>
I20260923 17:26:45.770268 16726 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:45.770360 16726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:45.770674 16726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:45.773814 16726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/data/instance:
uuid: "6b7cb688934a4bb086f4a514f59d0124"
format_stamp: "Formatted at 2026-09-23 17:26:45 on dist-test-slave-j0m4"
server_key: "5146175b0f40b557f90967d5b0bd062f"
server_key_iv: "81ecc5044bbbf73037ce360faef64492"
server_key_version: "encryptionkey@0"
I20260923 17:26:45.774677 16726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance:
uuid: "6b7cb688934a4bb086f4a514f59d0124"
format_stamp: "Formatted at 2026-09-23 17:26:45 on dist-test-slave-j0m4"
server_key: "5146175b0f40b557f90967d5b0bd062f"
server_key_iv: "81ecc5044bbbf73037ce360faef64492"
server_key_version: "encryptionkey@0"
I20260923 17:26:45.780309 16726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:26:45.784162 16740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:45.785696 16726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:26:45.785885 16726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/wal
uuid: "6b7cb688934a4bb086f4a514f59d0124"
format_stamp: "Formatted at 2026-09-23 17:26:45 on dist-test-slave-j0m4"
server_key: "5146175b0f40b557f90967d5b0bd062f"
server_key_iv: "81ecc5044bbbf73037ce360faef64492"
server_key_version: "encryptionkey@0"
I20260923 17:26:45.786114 16726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:45.805657 16726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:45.806687 16726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:45.807024 16726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:45.808228 16726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:45.809764 16726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:45.809896 16726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:45.810032 16726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:45.810122 16726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:45.843943 16726 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.195:36445
I20260923 17:26:45.843995 16852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.195:36445 every 8 connection(s)
I20260923 17:26:45.845328 16726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/data/info.pb
I20260923 17:26:45.849265 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 16726
I20260923 17:26:45.849488 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/wal/instance
I20260923 17:26:45.850044 28963 external_mini_cluster.cc:1428] Setting key 7b6c3d71256a9f7dd3234dff9a972c05
I20260923 17:26:45.859475 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:45.847993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.195:60509 (local address 127.28.72.254:37091)
0923 17:26:45.848205 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:45.848211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:45.850248 (+ 2037us) server_negotiation.cc:407] Connection header received
0923 17:26:45.852527 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:45.852532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:45.852635 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:45.852748 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:45.853915 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.854672 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.856529 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.856956 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.857109 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:45.858556 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:45.858561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:45.858565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:45.858584 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:45.858615 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:45.858627 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:45.858730 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:45.858995 (+ 265us) server_negotiation.cc:299] Negotiation successful
0923 17:26:45.859094 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:26:45.860553 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:45.848622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37091 (local address 127.28.72.195:60509)
0923 17:26:45.850053 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:45.850092 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:45.852236 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:45.853034 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:45.853046 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:45.853158 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:45.853748 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:45.853765 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.854852 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.854865 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:45.856388 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:45.856401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.857211 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.857218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:45.857462 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:45.858193 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:26:45.858196 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:45.858238 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:26:45.858324 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:45.858730 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:45.858743 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:45.858754 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 17:26:45.858944 (+ 190us) client_negotiation.cc:240] Negotiation successful
0923 17:26:45.859391 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"thread_start_us":641,"threads_started":1}
I20260923 17:26:45.862046 16853 heartbeater.cc:347] Connected to a master server at 127.28.72.254:37091
I20260923 17:26:45.862370 16853 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:45.863281 16853 heartbeater.cc:511] Master 127.28.72.254:37091 requested a full tablet report, sending...
I20260923 17:26:45.865175 16406 ts_manager.cc:194] Registered new tserver with Master: 6b7cb688934a4bb086f4a514f59d0124 (127.28.72.195:36445)
I20260923 17:26:45.866029 28963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:26:45.866302 16406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.72.195:60509
I20260923 17:26:45.877019 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:45.870706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.28.72.254:37091)
0923 17:26:45.870926 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:45.870933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:45.871087 (+ 154us) server_negotiation.cc:407] Connection header received
0923 17:26:45.871278 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:45.871283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:45.871345 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:45.871433 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:45.871961 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.872743 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.874144 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.874730 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.874863 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:45.876107 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:45.876112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:45.876115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:45.876136 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:45.876170 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:45.876181 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:45.876261 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:45.876556 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 17:26:45.876636 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 17:26:45.881708 16406 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:26:45.891703 16406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:26:45.913321 16592 heartbeater.cc:503] Master 127.28.72.254:37091 was elected leader, sending a full tablet report...
I20260923 17:26:45.920886 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:45.907143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.28.72.195:36445)
0923 17:26:45.912102 (+ 4959us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:45.912121 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:45.912159 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:26:45.912261 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:45.912269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:45.912513 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:45.912728 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:45.913486 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.915041 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.918065 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.919491 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.919670 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:45.919997 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:45.920272 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 17:26:45.920487 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4710,"thread_start_us":4494,"threads_started":1}
I20260923 17:26:45.920933 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:45.906716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36445 (local address 127.0.0.1:37170)
0923 17:26:45.907863 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:45.907928 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:45.908096 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:45.912738 (+ 4642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:45.912746 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:45.912818 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:45.913242 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:45.913256 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.915225 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.915234 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:45.917850 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:45.917864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.919699 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.919705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:45.919973 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:45.920082 (+ 109us) client_negotiation.cc:769] Sending connection context
0923 17:26:45.920227 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 17:26:45.920562 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":162,"thread_start_us":4030,"threads_started":1}
I20260923 17:26:45.922165 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:45.907191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:36331 (local address 127.0.0.1:50718)
0923 17:26:45.911191 (+ 4000us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:45.911218 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:45.911373 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:45.913301 (+ 1928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:45.913310 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:45.913353 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:45.913810 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:45.913827 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.916463 (+ 2636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.916468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:45.918876 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:45.918902 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.921329 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.921340 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:45.921567 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:45.921611 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 17:26:45.921710 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 17:26:45.921839 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3775,"thread_start_us":617,"threads_started":1}
I20260923 17:26:45.922564 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:45.907676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.28.72.194:36331)
0923 17:26:45.912453 (+ 4777us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:45.912468 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:45.912511 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:26:45.912643 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:45.912655 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:45.912992 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:45.914119 (+ 1127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:45.914459 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.916291 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.919167 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.920834 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.921122 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:45.921434 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:45.921782 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 17:26:45.922184 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4546,"thread_start_us":4410,"threads_started":1}
I20260923 17:26:45.925267 16788 tablet_service.cc:1467] Processing CreateTablet for tablet 635a782bc0cf4df28868044911e0354a (DEFAULT_TABLE table=test-table [id=9e52f6a2e91646649e3f6df18b54a7e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:45.926040 16658 tablet_service.cc:1467] Processing CreateTablet for tablet 635a782bc0cf4df28868044911e0354a (DEFAULT_TABLE table=test-table [id=9e52f6a2e91646649e3f6df18b54a7e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:45.926512 16788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635a782bc0cf4df28868044911e0354a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:45.927256 16658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635a782bc0cf4df28868044911e0354a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:45.933707 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:45.913715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.193:34441 (local address 127.0.0.1:49758)
0923 17:26:45.915694 (+ 1979us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:45.915714 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:45.915829 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:45.917212 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:45.917217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:45.917241 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:45.917700 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:45.917713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.919855 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.919861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:45.922699 (+ 2838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:45.922717 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.932687 (+ 9970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.932698 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:45.932926 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:45.932975 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 17:26:45.933093 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 17:26:45.933266 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1828,"mutex_wait_us":33,"thread_start_us":1676,"threads_started":1}
I20260923 17:26:45.933756 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:45.911725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.28.72.193:34441)
0923 17:26:45.916350 (+ 4625us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:45.916359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:45.916390 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:26:45.916494 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:45.916503 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:45.916759 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:45.916960 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:45.917913 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.919664 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.922934 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:45.932395 (+ 9461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:45.932583 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:45.932887 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:45.933191 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 17:26:45.933382 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4442,"thread_start_us":4994,"threads_started":1}
I20260923 17:26:45.938243 16527 tablet_service.cc:1467] Processing CreateTablet for tablet 635a782bc0cf4df28868044911e0354a (DEFAULT_TABLE table=test-table [id=9e52f6a2e91646649e3f6df18b54a7e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:45.939913 16527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635a782bc0cf4df28868044911e0354a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:45.941139 16871 tablet_bootstrap.cc:492] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124: Bootstrap starting.
I20260923 17:26:45.943931 16872 tablet_bootstrap.cc:492] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164: Bootstrap starting.
I20260923 17:26:45.945771 16871 tablet_bootstrap.cc:654] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:45.947750 16872 tablet_bootstrap.cc:654] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:45.947757 16871 log.cc:824] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:45.949138 16872 log.cc:824] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:45.951164 16871 tablet_bootstrap.cc:492] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124: No bootstrap required, opened a new log
I20260923 17:26:45.951650 16871 ts_tablet_manager.cc:1397] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 17:26:45.954248 16872 tablet_bootstrap.cc:492] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164: No bootstrap required, opened a new log
I20260923 17:26:45.954572 16872 ts_tablet_manager.cc:1397] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260923 17:26:45.956301 16875 tablet_bootstrap.cc:492] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540: Bootstrap starting.
I20260923 17:26:45.962054 16875 tablet_bootstrap.cc:654] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:45.963042 16872 raft_consensus.cc:348] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7cb688934a4bb086f4a514f59d0124" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36445 } } peers { permanent_uuid: "79100060896c4e2cbe665cdb2399e540" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34441 } } peers { permanent_uuid: "f4cbe347a56544abb2104ad935e56164" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36331 } }
I20260923 17:26:45.963539 16872 raft_consensus.cc:374] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:45.963639 16872 raft_consensus.cc:729] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4cbe347a56544abb2104ad935e56164, State: Initialized, Role: FOLLOWER
I20260923 17:26:45.963681 16875 log.cc:824] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:45.963816 16871 raft_consensus.cc:348] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7cb688934a4bb086f4a514f59d0124" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36445 } } peers { permanent_uuid: "79100060896c4e2cbe665cdb2399e540" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34441 } } peers { permanent_uuid: "f4cbe347a56544abb2104ad935e56164" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36331 } }
I20260923 17:26:45.964306 16871 raft_consensus.cc:374] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:45.964466 16871 raft_consensus.cc:729] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7cb688934a4bb086f4a514f59d0124, State: Initialized, Role: FOLLOWER
I20260923 17:26:45.964375 16872 consensus_queue.cc:260] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7cb688934a4bb086f4a514f59d0124" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36445 } } peers { permanent_uuid: "79100060896c4e2cbe665cdb2399e540" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34441 } } peers { permanent_uuid: "f4cbe347a56544abb2104ad935e56164" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36331 } }
I20260923 17:26:45.965210 16871 consensus_queue.cc:260] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7cb688934a4bb086f4a514f59d0124" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36445 } } peers { permanent_uuid: "79100060896c4e2cbe665cdb2399e540" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34441 } } peers { permanent_uuid: "f4cbe347a56544abb2104ad935e56164" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36331 } }
I20260923 17:26:45.966581 16853 heartbeater.cc:503] Master 127.28.72.254:37091 was elected leader, sending a full tablet report...
I20260923 17:26:45.967861 16723 heartbeater.cc:503] Master 127.28.72.254:37091 was elected leader, sending a full tablet report...
I20260923 17:26:45.968089 16875 tablet_bootstrap.cc:492] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540: No bootstrap required, opened a new log
I20260923 17:26:45.968626 16872 ts_tablet_manager.cc:1428] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260923 17:26:45.969216 16871 ts_tablet_manager.cc:1428] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260923 17:26:45.969655 16875 ts_tablet_manager.cc:1397] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260923 17:26:45.979038 16875 raft_consensus.cc:348] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7cb688934a4bb086f4a514f59d0124" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36445 } } peers { permanent_uuid: "79100060896c4e2cbe665cdb2399e540" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34441 } } peers { permanent_uuid: "f4cbe347a56544abb2104ad935e56164" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36331 } }
I20260923 17:26:45.979624 16875 raft_consensus.cc:374] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:45.979750 16875 raft_consensus.cc:729] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79100060896c4e2cbe665cdb2399e540, State: Initialized, Role: FOLLOWER
I20260923 17:26:45.980486 16875 consensus_queue.cc:260] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7cb688934a4bb086f4a514f59d0124" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36445 } } peers { permanent_uuid: "79100060896c4e2cbe665cdb2399e540" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34441 } } peers { permanent_uuid: "f4cbe347a56544abb2104ad935e56164" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36331 } }
I20260923 17:26:45.982616 16875 ts_tablet_manager.cc:1428] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260923 17:26:46.028046 16879 raft_consensus.cc:482] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:26:46.028481 16879 raft_consensus.cc:504] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7cb688934a4bb086f4a514f59d0124" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36445 } } peers { permanent_uuid: "79100060896c4e2cbe665cdb2399e540" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34441 } } peers { permanent_uuid: "f4cbe347a56544abb2104ad935e56164" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36331 } }
I20260923 17:26:46.030342 16879 leader_election.cc:288] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7cb688934a4bb086f4a514f59d0124 (127.28.72.195:36445), f4cbe347a56544abb2104ad935e56164 (127.28.72.194:36331)
I20260923 17:26:46.037956 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:46.030722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.195:36445 (local address 127.28.72.193:46117)
0923 17:26:46.031533 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:46.031556 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:46.031814 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:46.032373 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:46.032377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:46.032435 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:46.032824 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:46.032842 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.034027 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.034033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:46.035855 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:46.035869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.037189 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.037195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:46.037370 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:46.037403 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:26:46.037503 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 17:26:46.037604 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":89,"thread_start_us":1127,"threads_started":1}
I20260923 17:26:46.038004 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:46.030955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:46117 (local address 127.28.72.195:36445)
0923 17:26:46.031252 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:46.031259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:46.031653 (+ 394us) server_negotiation.cc:407] Connection header received
0923 17:26:46.032016 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:46.032029 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:46.032096 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:46.032149 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:46.033001 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.033874 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.036088 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.036995 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.037137 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:46.037191 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:46.037581 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 17:26:46.037727 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260923 17:26:46.038352 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:46.030722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.194:36331 (local address 127.28.72.193:43893)
0923 17:26:46.032170 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:46.032190 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:46.032332 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:46.032725 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:46.032729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:46.032757 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:46.033210 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:46.033222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.034551 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.034555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:46.036164 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:46.036178 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.037463 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.037468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:46.037682 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:46.037715 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:26:46.037816 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 17:26:46.037951 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330,"mutex_wait_us":50,"thread_start_us":609,"threads_started":1}
I20260923 17:26:46.039062 16808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635a782bc0cf4df28868044911e0354a" candidate_uuid: "79100060896c4e2cbe665cdb2399e540" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7cb688934a4bb086f4a514f59d0124" is_pre_election: true
I20260923 17:26:46.039270 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:46.030990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.72.193:43893 (local address 127.28.72.194:36331)
0923 17:26:46.031436 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:46.031442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:46.032309 (+ 867us) server_negotiation.cc:407] Connection header received
0923 17:26:46.032492 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:46.032499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:46.032597 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:46.032699 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:46.033471 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.034371 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.036371 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.037187 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.037344 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:46.037421 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:46.037866 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 17:26:46.038023 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299}
I20260923 17:26:46.039705 16808 raft_consensus.cc:2454] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79100060896c4e2cbe665cdb2399e540 in term 0.
I20260923 17:26:46.040521 16483 leader_election.cc:302] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7cb688934a4bb086f4a514f59d0124, 79100060896c4e2cbe665cdb2399e540; no voters:
I20260923 17:26:46.040458 16678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635a782bc0cf4df28868044911e0354a" candidate_uuid: "79100060896c4e2cbe665cdb2399e540" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4cbe347a56544abb2104ad935e56164" is_pre_election: true
I20260923 17:26:46.040998 16879 raft_consensus.cc:2781] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:26:46.041035 16678 raft_consensus.cc:2454] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79100060896c4e2cbe665cdb2399e540 in term 0.
I20260923 17:26:46.041175 16879 raft_consensus.cc:482] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:26:46.041282 16879 raft_consensus.cc:3037] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:46.043622 16879 raft_consensus.cc:504] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7cb688934a4bb086f4a514f59d0124" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36445 } } peers { permanent_uuid: "79100060896c4e2cbe665cdb2399e540" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34441 } } peers { permanent_uuid: "f4cbe347a56544abb2104ad935e56164" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36331 } }
I20260923 17:26:46.044234 16879 leader_election.cc:288] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7cb688934a4bb086f4a514f59d0124 (127.28.72.195:36445), f4cbe347a56544abb2104ad935e56164 (127.28.72.194:36331)
I20260923 17:26:46.044816 16678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635a782bc0cf4df28868044911e0354a" candidate_uuid: "79100060896c4e2cbe665cdb2399e540" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4cbe347a56544abb2104ad935e56164"
I20260923 17:26:46.044901 16808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635a782bc0cf4df28868044911e0354a" candidate_uuid: "79100060896c4e2cbe665cdb2399e540" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7cb688934a4bb086f4a514f59d0124"
I20260923 17:26:46.045012 16678 raft_consensus.cc:3037] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:46.045219 16808 raft_consensus.cc:3037] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:46.047518 16678 raft_consensus.cc:2454] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79100060896c4e2cbe665cdb2399e540 in term 1.
I20260923 17:26:46.048053 16482 leader_election.cc:302] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79100060896c4e2cbe665cdb2399e540, f4cbe347a56544abb2104ad935e56164; no voters:
I20260923 17:26:46.048126 16808 raft_consensus.cc:2454] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79100060896c4e2cbe665cdb2399e540 in term 1.
I20260923 17:26:46.048447 16879 raft_consensus.cc:2781] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:46.049110 16879 raft_consensus.cc:686] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [term 1 LEADER]: Becoming Leader. State: Replica: 79100060896c4e2cbe665cdb2399e540, State: Running, Role: LEADER
I20260923 17:26:46.049618 16879 consensus_queue.cc:237] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7cb688934a4bb086f4a514f59d0124" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36445 } } peers { permanent_uuid: "79100060896c4e2cbe665cdb2399e540" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34441 } } peers { permanent_uuid: "f4cbe347a56544abb2104ad935e56164" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36331 } }
I20260923 17:26:46.054668 16405 catalog_manager.cc:5495] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79100060896c4e2cbe665cdb2399e540 (127.28.72.193). New cstate: current_term: 1 leader_uuid: "79100060896c4e2cbe665cdb2399e540" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7cb688934a4bb086f4a514f59d0124" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79100060896c4e2cbe665cdb2399e540" member_type: VOTER last_known_addr { host: "127.28.72.193" port: 34441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4cbe347a56544abb2104ad935e56164" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36331 } health_report { overall_health: UNKNOWN } } }
I20260923 17:26:46.077060 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:46.070252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.28.72.193:34441)
0923 17:26:46.070491 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:46.070496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:46.070517 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:26:46.070676 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:46.070680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:46.070793 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:46.070932 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:26:46.071684 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.072680 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.074307 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.074743 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.074862 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:46.075625 (+ 763us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:26:46.076199 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:26:46.076342 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:46.076684 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 17:26:46.076784 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:26:46.087433 16808 raft_consensus.cc:1261] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124 [term 1 FOLLOWER]: Refusing update from remote peer 79100060896c4e2cbe665cdb2399e540: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:26:46.087540 16678 raft_consensus.cc:1261] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164 [term 1 FOLLOWER]: Refusing update from remote peer 79100060896c4e2cbe665cdb2399e540: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:26:46.088709 16882 consensus_queue.cc:1035] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7cb688934a4bb086f4a514f59d0124" member_type: VOTER last_known_addr { host: "127.28.72.195" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:26:46.089126 16879 consensus_queue.cc:1035] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4cbe347a56544abb2104ad935e56164" member_type: VOTER last_known_addr { host: "127.28.72.194" port: 36331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:26:46.097002 16854 tablet.cc:2369] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:26:46.125483 16406 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:26:46.125833 16406 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:26:46.130029 16406 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33adf2ff87d7453481fb9c5bbb95d4e5: Sending DeleteTablet for 3 replicas of tablet 635a782bc0cf4df28868044911e0354a
I20260923 17:26:46.131470 16788 tablet_service.cc:1514] Processing DeleteTablet for tablet 635a782bc0cf4df28868044911e0354a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:46 UTC) from {username='slave'} at 127.0.0.1:37170
I20260923 17:26:46.131470 16658 tablet_service.cc:1514] Processing DeleteTablet for tablet 635a782bc0cf4df28868044911e0354a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:46 UTC) from {username='slave'} at 127.0.0.1:50718
I20260923 17:26:46.131584 16527 tablet_service.cc:1514] Processing DeleteTablet for tablet 635a782bc0cf4df28868044911e0354a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:46 UTC) from {username='slave'} at 127.0.0.1:49758
I20260923 17:26:46.138487 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 16465
I20260923 17:26:46.145716 16896 tablet_replica.cc:330] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124: stopping tablet replica
I20260923 17:26:46.146559 16896 raft_consensus.cc:2229] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:26:46.147440 16896 raft_consensus.cc:2258] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:46.150817 16896 ts_tablet_manager.cc:1905] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:46.161706 16896 ts_tablet_manager.cc:1918] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:46.162118 16896 log.cc:1196] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-2/wal/wals/635a782bc0cf4df28868044911e0354a
I20260923 17:26:46.162926 16896 ts_tablet_manager.cc:1939] T 635a782bc0cf4df28868044911e0354a P 6b7cb688934a4bb086f4a514f59d0124: Deleting consensus metadata
I20260923 17:26:46.165722 16393 catalog_manager.cc:4841] TS 6b7cb688934a4bb086f4a514f59d0124 (127.28.72.195:36445): tablet 635a782bc0cf4df28868044911e0354a (table test-table [id=9e52f6a2e91646649e3f6df18b54a7e0]) successfully deleted
I20260923 17:26:46.167908 16898 tablet_replica.cc:330] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540: stopping tablet replica
I20260923 17:26:46.168792 16898 raft_consensus.cc:2229] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:26:46.169914 16898 raft_consensus.cc:2258] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:46.170872 16897 tablet_replica.cc:330] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164: stopping tablet replica
I20260923 17:26:46.171362 16897 raft_consensus.cc:2229] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:26:46.171864 16897 raft_consensus.cc:2258] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:46.172617 16898 ts_tablet_manager.cc:1905] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:46.173539 16897 ts_tablet_manager.cc:1905] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:26:46.178395 16897 ts_tablet_manager.cc:1918] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:46.178610 16897 log.cc:1196] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-1/wal/wals/635a782bc0cf4df28868044911e0354a
I20260923 17:26:46.179613 16897 ts_tablet_manager.cc:1939] T 635a782bc0cf4df28868044911e0354a P f4cbe347a56544abb2104ad935e56164: Deleting consensus metadata
I20260923 17:26:46.180703 16898 ts_tablet_manager.cc:1918] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:46.180920 16898 log.cc:1196] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540: Deleting WAL directory at /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184070511153-28963-0/minicluster-data/ts-0/wal/wals/635a782bc0cf4df28868044911e0354a
I20260923 17:26:46.181489 16898 ts_tablet_manager.cc:1939] T 635a782bc0cf4df28868044911e0354a P 79100060896c4e2cbe665cdb2399e540: Deleting consensus metadata
I20260923 17:26:46.181725 16392 catalog_manager.cc:4841] TS f4cbe347a56544abb2104ad935e56164 (127.28.72.194:36331): tablet 635a782bc0cf4df28868044911e0354a (table test-table [id=9e52f6a2e91646649e3f6df18b54a7e0]) successfully deleted
I20260923 17:26:46.182719 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:46.140263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.28.72.193:34441)
0923 17:26:46.140475 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:46.140482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:46.163970 (+ 23488us) server_negotiation.cc:407] Connection header received
0923 17:26:46.164514 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:46.164522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:46.164654 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:46.164744 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:46.165451 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.166741 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.168849 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.169566 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.169678 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:46.179888 (+ 10210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:46.179915 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:46.179935 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:46.180995 (+ 1060us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:46.181065 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:46.181103 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:46.181228 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:46.182187 (+ 959us) server_negotiation.cc:299] Negotiation successful
0923 17:26:46.182304 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:26:46.183561 16587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:46.183893 16393 catalog_manager.cc:4841] TS 79100060896c4e2cbe665cdb2399e540 (127.28.72.193:34441): tablet 635a782bc0cf4df28868044911e0354a (table test-table [id=9e52f6a2e91646649e3f6df18b54a7e0]) successfully deleted
I20260923 17:26:46.288908 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 16465
I20260923 17:26:46.308473 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 16596
I20260923 17:26:46.317019 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:46.309424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.28.72.194:36331)
0923 17:26:46.309599 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:46.309603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:46.309624 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:26:46.309885 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:46.309890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:46.309980 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:46.310070 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:46.310594 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.311492 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.313648 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.314096 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.314204 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:46.315548 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:46.315558 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:46.315574 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:46.316180 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:46.316234 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:46.316256 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:46.316352 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:46.316540 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 17:26:46.316643 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:26:46.317914 16718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:46.408812 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 16596
I20260923 17:26:46.426342 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 16726
I20260923 17:26:46.434252 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:46.427199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.28.72.195:36445)
0923 17:26:46.427446 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:46.427454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:46.427589 (+ 135us) server_negotiation.cc:407] Connection header received
0923 17:26:46.427832 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:46.427837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:46.427951 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:46.428072 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:46.428509 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.429409 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.430852 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:46.431355 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:46.431503 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:46.432812 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:46.432822 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:46.432839 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:46.433427 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:46.433478 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:46.433497 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:46.433636 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:46.433815 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 17:26:46.433943 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:26:46.434895 16848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:46.527675 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 16726
I20260923 17:26:46.545027 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 16375
I20260923 17:26:46.546360 16436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:46.665087 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 16375
2026-09-23T17:26:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2894 ms)
[----------] 6 tests from EncryptionPolicyTest (22772 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:26:46 dist-test-slave-j0m4 krb5kdc[16906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:26:46 dist-test-slave-j0m4 krb5kdc[16906](info): set up 2 sockets
Sep 23 17:26:46 dist-test-slave-j0m4 krb5kdc[16906](info): commencing operation
krb5kdc: starting...
W20260923 17:26:47.807809 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:26:47 dist-test-slave-j0m4 krb5kdc[16906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:26:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:26:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:26:48.087298 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33915
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:48.436743 16919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:48.437111 16919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:48.437289 16919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:48.444684 16919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:26:48.444871 16919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:48.444990 16919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:48.445096 16919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:26:48.445184 16919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:26:48.455404 16919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33915
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:48.456851 16919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:48.458230 16919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:48.470664 16927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:48.470862 16924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:48.470835 16925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:48.472321 16919 server_base.cc:1034] running on GCE node
I20260923 17:26:48.473320 16919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:48.475036 16919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:48.476366 16919 hybrid_clock.cc:648] HybridClock initialized: now 1790184408476292 us; error 56 us; skew 500 ppm
I20260923 17:26:48.476864 16919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:48 dist-test-slave-j0m4 krb5kdc[16906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:48.480818 16919 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:26:48.482725 16919 webserver.cc:460] Webserver started at http://127.0.0.1:35083/ using document root <none> and password file <none>
I20260923 17:26:48.483609 16919 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:48.483776 16919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:48.484118 16919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:48.487452 16919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "6397cbabd53b4213ad33b3bf3e1eccd4"
format_stamp: "Formatted at 2026-09-23 17:26:48 on dist-test-slave-j0m4"
server_key: "dfee3430683275bed82e73007b4ff35c"
server_key_iv: "46c40125ba34e0b9ee54474163cf01fb"
server_key_version: "encryptionkey@0"
I20260923 17:26:48.488313 16919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "6397cbabd53b4213ad33b3bf3e1eccd4"
format_stamp: "Formatted at 2026-09-23 17:26:48 on dist-test-slave-j0m4"
server_key: "dfee3430683275bed82e73007b4ff35c"
server_key_iv: "46c40125ba34e0b9ee54474163cf01fb"
server_key_version: "encryptionkey@0"
I20260923 17:26:48.494138 16919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:26:48.498394 16934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:48.500129 16919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:26:48.500434 16919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "6397cbabd53b4213ad33b3bf3e1eccd4"
format_stamp: "Formatted at 2026-09-23 17:26:48 on dist-test-slave-j0m4"
server_key: "dfee3430683275bed82e73007b4ff35c"
server_key_iv: "46c40125ba34e0b9ee54474163cf01fb"
server_key_version: "encryptionkey@0"
I20260923 17:26:48.500674 16919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:48.514816 16919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:48.519752 16919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:48.520102 16919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:48.540055 16919 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33915
I20260923 17:26:48.540091 16985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33915 every 8 connection(s)
I20260923 17:26:48.541563 16919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:26:48.544504 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 16919
I20260923 17:26:48.544719 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:26:48.545538 28963 external_mini_cluster.cc:1428] Setting key f5c41e1a42185f94f204592a5165d976
I20260923 17:26:48.546474 16986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:26:48 dist-test-slave-j0m4 krb5kdc[16906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:26:48.562525 16986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4: Bootstrap starting.
I20260923 17:26:48.567188 16986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:48.568567 16986 log.cc:824] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:48.571879 16986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4: No bootstrap required, opened a new log
I20260923 17:26:48.572018 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:48.547767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34357 (local address 127.0.0.1:33915)
0923 17:26:48.548703 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:48.548715 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:48.548757 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:26:48.550344 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:48.550369 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:48.550827 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:48.551283 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:48.553214 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:48.554646 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:48.556549 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:48.557279 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:48.557493 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:48.562667 (+ 5174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:48.562681 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:48.562703 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:48.562767 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:48.566889 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:48.568049 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:48.568058 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:48.568067 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:48.568314 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:48.568918 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:48.568922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:48.568925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:48.569575 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:48.569991 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:48.570502 (+ 511us) server_negotiation.cc:299] Negotiation successful
0923 17:26:48.570849 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":441,"threads_started":1}
I20260923 17:26:48.576682 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:26:48.580878 16986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6397cbabd53b4213ad33b3bf3e1eccd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33915 } }
I20260923 17:26:48.581369 16986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:48.581528 16986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6397cbabd53b4213ad33b3bf3e1eccd4, State: Initialized, Role: FOLLOWER
I20260923 17:26:48.582244 16986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6397cbabd53b4213ad33b3bf3e1eccd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33915 } }
I20260923 17:26:48.582501 16986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:26:48.582651 16986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:26:48.582865 16986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:48.585978 16986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6397cbabd53b4213ad33b3bf3e1eccd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33915 } }
I20260923 17:26:48.586612 16986 leader_election.cc:302] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6397cbabd53b4213ad33b3bf3e1eccd4; no voters:
I20260923 17:26:48.587375 16986 leader_election.cc:288] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:26:48.587502 16996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:48.589859 16986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:26:48.591485 16996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [term 1 LEADER]: Becoming Leader. State: Replica: 6397cbabd53b4213ad33b3bf3e1eccd4, State: Running, Role: LEADER
I20260923 17:26:48.592065 16996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6397cbabd53b4213ad33b3bf3e1eccd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33915 } }
I20260923 17:26:48.596113 16997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6397cbabd53b4213ad33b3bf3e1eccd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6397cbabd53b4213ad33b3bf3e1eccd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33915 } } }
I20260923 17:26:48.596508 16998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6397cbabd53b4213ad33b3bf3e1eccd4. Latest consensus state: current_term: 1 leader_uuid: "6397cbabd53b4213ad33b3bf3e1eccd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6397cbabd53b4213ad33b3bf3e1eccd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33915 } } }
I20260923 17:26:48.597424 16997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:48.597474 16998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:48.606752 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:48.581636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59059 (local address 127.0.0.1:33915)
0923 17:26:48.581851 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:48.581862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:48.582439 (+ 577us) server_negotiation.cc:407] Connection header received
0923 17:26:48.582655 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:48.582663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:48.582732 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:48.582861 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:48.584747 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:48.585503 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:48.587363 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:48.588330 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:48.589993 (+ 1663us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:48.594319 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:48.594328 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:48.594333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:48.594383 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:48.600647 (+ 6264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:48.602117 (+ 1470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:48.602126 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:48.602132 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:48.602399 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:48.603189 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:48.603197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:48.603203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:48.603760 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:48.605671 (+ 1911us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:48.605910 (+ 239us) server_negotiation.cc:299] Negotiation successful
0923 17:26:48.606170 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:26:48.610519 17004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:26:48.616855 17004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:26:48.626981 17004 catalog_manager.cc:1347] Generated new cluster ID: 28dfb5a7d2d64fa5aa4f011949968582
I20260923 17:26:48.627146 17004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:26:48.640847 17004 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:26:48.642370 17004 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:26:48.652380 17004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6397cbabd53b4213ad33b3bf3e1eccd4: Generated new TSK 0
I20260923 17:26:48.653565 17004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:26:48.671023 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 16919
I20260923 17:26:48.673029 16981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:48.770358 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 16919
W20260923 17:26:48.788785 16904 connection.cc:535] client connection to 127.0.0.1:33915 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33915): Connection reset by peer (error 104)
2026-09-23T17:26:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2099 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:26:48 dist-test-slave-j0m4 krb5kdc[17018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:26:48 dist-test-slave-j0m4 krb5kdc[17018](info): set up 2 sockets
Sep 23 17:26:48 dist-test-slave-j0m4 krb5kdc[17018](info): commencing operation
krb5kdc: starting...
W20260923 17:26:49.888197 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:26:50 dist-test-slave-j0m4 krb5kdc[17018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:26:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:26:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:26:50.175009 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:50.520501 17031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:50.520885 17031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:50.521065 17031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:50.528949 17031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:26:50.529160 17031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:50.529294 17031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:50.529399 17031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:26:50.529484 17031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:26:50.540282 17031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:50.541738 17031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:50.543218 17031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:50.556237 17036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:50.556137 17039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:50.556206 17037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:50.557190 17031 server_base.cc:1034] running on GCE node
I20260923 17:26:50.558130 17031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:50.559983 17031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:50.561359 17031 hybrid_clock.cc:648] HybridClock initialized: now 1790184410561233 us; error 45 us; skew 500 ppm
I20260923 17:26:50.561834 17031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:50 dist-test-slave-j0m4 krb5kdc[17018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:50.565533 17031 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:26:50.567425 17031 webserver.cc:460] Webserver started at http://127.0.0.1:43223/ using document root <none> and password file <none>
I20260923 17:26:50.568193 17031 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:50.568308 17031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:50.568596 17031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:50.572126 17031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "ea448198767d4eaca8e115b194505f18"
format_stamp: "Formatted at 2026-09-23 17:26:50 on dist-test-slave-j0m4"
server_key: "114ef9a2e2003ce5419ed31862fc3a73"
server_key_iv: "2f31362a3c386f6e558be3e4e2a6a001"
server_key_version: "encryptionkey@0"
I20260923 17:26:50.573069 17031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "ea448198767d4eaca8e115b194505f18"
format_stamp: "Formatted at 2026-09-23 17:26:50 on dist-test-slave-j0m4"
server_key: "114ef9a2e2003ce5419ed31862fc3a73"
server_key_iv: "2f31362a3c386f6e558be3e4e2a6a001"
server_key_version: "encryptionkey@0"
I20260923 17:26:50.579327 17031 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 17:26:50.583855 17046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:50.585544 17031 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:26:50.585758 17031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "ea448198767d4eaca8e115b194505f18"
format_stamp: "Formatted at 2026-09-23 17:26:50 on dist-test-slave-j0m4"
server_key: "114ef9a2e2003ce5419ed31862fc3a73"
server_key_iv: "2f31362a3c386f6e558be3e4e2a6a001"
server_key_version: "encryptionkey@0"
I20260923 17:26:50.586198 17031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:50.614478 17031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:50.618805 17031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:50.619171 17031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:50.638463 17031 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39945
I20260923 17:26:50.638482 17097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39945 every 8 connection(s)
I20260923 17:26:50.640100 17031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:26:50.642426 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 17031
I20260923 17:26:50.642614 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:26:50.643074 28963 external_mini_cluster.cc:1428] Setting key 3b64d388c82a16cf6bb4f93248d61059
I20260923 17:26:50.644622 17098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:50.656512 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:50.644968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59387 (local address 127.0.0.1:39945)
0923 17:26:50.645761 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:50.645773 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:50.645814 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:26:50.647563 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:50.647588 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:50.648017 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:50.648434 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:50.648990 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:50.650314 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:50.651783 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:50.652474 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:50.652681 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:50.654033 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:50.654042 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:50.654059 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:50.654677 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:50.654760 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:50.654792 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:50.654974 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:50.655332 (+ 358us) server_negotiation.cc:299] Negotiation successful
0923 17:26:50.655511 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":336,"threads_started":1}
I20260923 17:26:50.657184 17098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18: Bootstrap starting.
I20260923 17:26:50.660580 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:26:50.660786 17098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:50.661882 17098 log.cc:824] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:50.665628 17098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18: No bootstrap required, opened a new log
I20260923 17:26:50.672961 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:50.665481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41403 (local address 127.0.0.1:39945)
0923 17:26:50.665674 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:50.665681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:50.666121 (+ 440us) server_negotiation.cc:407] Connection header received
0923 17:26:50.666435 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:50.666439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:50.666509 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:50.666607 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:50.667249 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:50.668114 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:50.670084 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:50.670528 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:50.670627 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:50.672045 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:50.672050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:50.672053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:50.672075 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:50.672118 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:50.672144 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:50.672246 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:50.672503 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 17:26:50.672589 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:26:50.675848 17098 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea448198767d4eaca8e115b194505f18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260923 17:26:50.676303 17098 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:50.676429 17098 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea448198767d4eaca8e115b194505f18, State: Initialized, Role: FOLLOWER
I20260923 17:26:50.677040 17098 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea448198767d4eaca8e115b194505f18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260923 17:26:50.677276 17098 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:26:50.677440 17098 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:26:50.677620 17098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:50.679993 17098 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea448198767d4eaca8e115b194505f18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260923 17:26:50.680558 17098 leader_election.cc:302] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea448198767d4eaca8e115b194505f18; no voters:
I20260923 17:26:50.681283 17098 leader_election.cc:288] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:26:50.681394 17108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:50.682199 17108 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [term 1 LEADER]: Becoming Leader. State: Replica: ea448198767d4eaca8e115b194505f18, State: Running, Role: LEADER
I20260923 17:26:50.682744 17108 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea448198767d4eaca8e115b194505f18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260923 17:26:50.683476 17098 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:26:50.685859 17109 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea448198767d4eaca8e115b194505f18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea448198767d4eaca8e115b194505f18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } } }
I20260923 17:26:50.686261 17109 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:50.686826 17110 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea448198767d4eaca8e115b194505f18. Latest consensus state: current_term: 1 leader_uuid: "ea448198767d4eaca8e115b194505f18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea448198767d4eaca8e115b194505f18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } } }
I20260923 17:26:50.687217 17110 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:50.689570 17116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:26:50.694702 17116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:26:50.704564 17116 catalog_manager.cc:1347] Generated new cluster ID: 938d439711aa4b3297ed1f2d3eb3b8f0
I20260923 17:26:50.704723 17116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:26:50.718271 17116 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:26:50.720538 17116 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:26:50.733865 17116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea448198767d4eaca8e115b194505f18: Generated new TSK 0
I20260923 17:26:50.735076 17116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:26:50.752467 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17031
W20260923 17:26:50.753780 17093 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59387
W20260923 17:26:50.754703 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17031: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:26:50.754927 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 17031
W20260923 17:26:50.773119 17016 connection.cc:535] client connection to 127.0.0.1:39945 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39945): Connection reset by peer (error 104)
2026-09-23T17:26:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1983 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:26:50 dist-test-slave-j0m4 krb5kdc[17129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:26:50 dist-test-slave-j0m4 krb5kdc[17129](info): set up 2 sockets
Sep 23 17:26:50 dist-test-slave-j0m4 krb5kdc[17129](info): commencing operation
krb5kdc: starting...
W20260923 17:26:51.893460 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:26:52 dist-test-slave-j0m4 krb5kdc[17129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:26:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:26:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:26:52.167459 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:52.516605 17142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:52.516918 17142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:52.517014 17142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:52.524324 17142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:26:52.524462 17142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:52.524511 17142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:52.524570 17142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:26:52.524616 17142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:26:52.534785 17142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41913
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:52.536247 17142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:52.537619 17142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:52.550788 17147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:52.550875 17150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:52.550848 17148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:52.552040 17142 server_base.cc:1034] running on GCE node
I20260923 17:26:52.552991 17142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:52.554598 17142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:52.555912 17142 hybrid_clock.cc:648] HybridClock initialized: now 1790184412555846 us; error 48 us; skew 500 ppm
I20260923 17:26:52.556458 17142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:52 dist-test-slave-j0m4 krb5kdc[17129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:52.564785 17142 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:26:52.566586 17142 webserver.cc:460] Webserver started at http://127.0.0.1:34889/ using document root <none> and password file <none>
I20260923 17:26:52.567451 17142 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:52.567596 17142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:52.567950 17142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:52.571288 17142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "a962c20554814ce8856ae2c441eb33fe"
format_stamp: "Formatted at 2026-09-23 17:26:52 on dist-test-slave-j0m4"
server_key: "76454c349eadd6bf91fc55122cba45a3"
server_key_iv: "2144098db6d03771a96ae396ac0cf6e9"
server_key_version: "encryptionkey@0"
I20260923 17:26:52.572168 17142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "a962c20554814ce8856ae2c441eb33fe"
format_stamp: "Formatted at 2026-09-23 17:26:52 on dist-test-slave-j0m4"
server_key: "76454c349eadd6bf91fc55122cba45a3"
server_key_iv: "2144098db6d03771a96ae396ac0cf6e9"
server_key_version: "encryptionkey@0"
I20260923 17:26:52.578332 17142 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260923 17:26:52.582645 17157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:52.584457 17142 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260923 17:26:52.584671 17142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "a962c20554814ce8856ae2c441eb33fe"
format_stamp: "Formatted at 2026-09-23 17:26:52 on dist-test-slave-j0m4"
server_key: "76454c349eadd6bf91fc55122cba45a3"
server_key_iv: "2144098db6d03771a96ae396ac0cf6e9"
server_key_version: "encryptionkey@0"
I20260923 17:26:52.584901 17142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:52.591038 17142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:52.595757 17142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:52.596071 17142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:52.615799 17142 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41913
I20260923 17:26:52.615839 17208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41913 every 8 connection(s)
I20260923 17:26:52.617341 17142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:26:52.621495 17209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:52.624675 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 17142
I20260923 17:26:52.624872 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:26:52.625319 28963 external_mini_cluster.cc:1428] Setting key 5c6f661eb487fc95bbd67f3806906f89
I20260923 17:26:52.633551 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:52.627279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38135 (local address 127.0.0.1:41913)
0923 17:26:52.628086 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:52.628099 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:52.628141 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:26:52.629827 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:52.629853 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:52.630241 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:52.630667 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:52.631859 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:52.631865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:52.631877 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:52.631894 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:52.631952 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:52.631971 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:52.632231 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:52.632384 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 17:26:52.632584 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":330,"threads_started":1}
I20260923 17:26:52.633584 17209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe: Bootstrap starting.
I20260923 17:26:52.637298 17209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:52.637683 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:26:52.638481 17209 log.cc:824] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:52.641848 17209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe: No bootstrap required, opened a new log
I20260923 17:26:52.646189 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:52.643493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60453 (local address 127.0.0.1:41913)
0923 17:26:52.643670 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:52.643675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:52.644155 (+ 480us) server_negotiation.cc:407] Connection header received
0923 17:26:52.644373 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:52.644377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:52.644437 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:52.644587 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:52.645522 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:52.645527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:52.645530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:52.645549 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:52.645588 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:52.645602 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:52.645742 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:52.645840 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 17:26:52.645909 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:26:52.650990 17209 raft_consensus.cc:348] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a962c20554814ce8856ae2c441eb33fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41913 } }
I20260923 17:26:52.651486 17209 raft_consensus.cc:374] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:52.651604 17209 raft_consensus.cc:729] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962c20554814ce8856ae2c441eb33fe, State: Initialized, Role: FOLLOWER
I20260923 17:26:52.652278 17209 consensus_queue.cc:260] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a962c20554814ce8856ae2c441eb33fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41913 } }
I20260923 17:26:52.652611 17209 raft_consensus.cc:388] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:26:52.652765 17209 raft_consensus.cc:482] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:26:52.652972 17209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:52.655402 17209 raft_consensus.cc:504] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a962c20554814ce8856ae2c441eb33fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41913 } }
I20260923 17:26:52.655952 17209 leader_election.cc:302] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a962c20554814ce8856ae2c441eb33fe; no voters:
I20260923 17:26:52.656607 17209 leader_election.cc:288] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:26:52.656773 17219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:52.657467 17219 raft_consensus.cc:686] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [term 1 LEADER]: Becoming Leader. State: Replica: a962c20554814ce8856ae2c441eb33fe, State: Running, Role: LEADER
I20260923 17:26:52.658113 17219 consensus_queue.cc:237] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a962c20554814ce8856ae2c441eb33fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41913 } }
I20260923 17:26:52.658900 17209 sys_catalog.cc:564] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:26:52.661528 17221 sys_catalog.cc:455] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a962c20554814ce8856ae2c441eb33fe. Latest consensus state: current_term: 1 leader_uuid: "a962c20554814ce8856ae2c441eb33fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a962c20554814ce8856ae2c441eb33fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41913 } } }
I20260923 17:26:52.661986 17221 sys_catalog.cc:458] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:52.662730 17220 sys_catalog.cc:455] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a962c20554814ce8856ae2c441eb33fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a962c20554814ce8856ae2c441eb33fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41913 } } }
I20260923 17:26:52.663040 17220 sys_catalog.cc:458] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:52.663805 17228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:26:52.669855 17228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:26:52.680636 17228 catalog_manager.cc:1347] Generated new cluster ID: 7ae76e5f31424c1c9313970d3c2acd8b
I20260923 17:26:52.680802 17228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:26:52.701645 17228 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:26:52.702205 17228 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:26:52.711002 17228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a962c20554814ce8856ae2c441eb33fe: Generated new TSK 0
I20260923 17:26:52.712013 17228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:26:52.720590 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17142
W20260923 17:26:52.721843 17204 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38135
W20260923 17:26:52.722592 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17142: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:26:52.722740 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 17142
2026-09-23T17:26:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:26:52 dist-test-slave-j0m4 krb5kdc[17240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:26:52 dist-test-slave-j0m4 krb5kdc[17240](info): set up 2 sockets
Sep 23 17:26:52 dist-test-slave-j0m4 krb5kdc[17240](info): commencing operation
krb5kdc: starting...
W20260923 17:26:53.839749 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.009s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:26:54 dist-test-slave-j0m4 krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:26:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:26:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:26:54.114979 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:54.462277 17253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:54.462677 17253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:54.462858 17253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:54.470523 17253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:26:54.470724 17253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:54.470840 17253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:54.470960 17253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:26:54.471032 17253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:26:54.481338 17253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:54.482817 17253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:54.484270 17253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:54.497020 17261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:54.497102 17259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:54.497102 17258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:54.498111 17253 server_base.cc:1034] running on GCE node
I20260923 17:26:54.499069 17253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:54.500713 17253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:54.501973 17253 hybrid_clock.cc:648] HybridClock initialized: now 1790184414501913 us; error 53 us; skew 500 ppm
I20260923 17:26:54.502359 17253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:54 dist-test-slave-j0m4 krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:54.505822 17253 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:26:54.507612 17253 webserver.cc:460] Webserver started at http://127.0.0.1:44829/ using document root <none> and password file <none>
I20260923 17:26:54.508368 17253 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:54.508543 17253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:54.508868 17253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:54.512202 17253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "c142334c6b0e4ef994356563ae4f72dd"
format_stamp: "Formatted at 2026-09-23 17:26:54 on dist-test-slave-j0m4"
server_key: "e5c202669be76fd01b23d98a12c4b251"
server_key_iv: "5d85f20af79f9a9ef459ab60e4b7dfba"
server_key_version: "encryptionkey@0"
I20260923 17:26:54.513090 17253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "c142334c6b0e4ef994356563ae4f72dd"
format_stamp: "Formatted at 2026-09-23 17:26:54 on dist-test-slave-j0m4"
server_key: "e5c202669be76fd01b23d98a12c4b251"
server_key_iv: "5d85f20af79f9a9ef459ab60e4b7dfba"
server_key_version: "encryptionkey@0"
I20260923 17:26:54.518898 17253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:26:54.522787 17268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:54.524390 17253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:26:54.524650 17253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "c142334c6b0e4ef994356563ae4f72dd"
format_stamp: "Formatted at 2026-09-23 17:26:54 on dist-test-slave-j0m4"
server_key: "e5c202669be76fd01b23d98a12c4b251"
server_key_iv: "5d85f20af79f9a9ef459ab60e4b7dfba"
server_key_version: "encryptionkey@0"
I20260923 17:26:54.524873 17253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:54.547184 17253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:54.551985 17253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:54.552326 17253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:54.572826 17253 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38569
I20260923 17:26:54.572876 17319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38569 every 8 connection(s)
I20260923 17:26:54.574357 17253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:26:54.578706 17320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:54.583029 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 17253
I20260923 17:26:54.583273 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:26:54.583812 28963 external_mini_cluster.cc:1428] Setting key cfe8284cb1cd45fa3109f3a038ee987b
I20260923 17:26:54.592253 17320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd: Bootstrap starting.
I20260923 17:26:54.595611 17320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:54.597152 17320 log.cc:824] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:26:54 dist-test-slave-j0m4 krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:26:54.600803 17320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd: No bootstrap required, opened a new log
I20260923 17:26:54.608893 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:54.585788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49459 (local address 127.0.0.1:38569)
0923 17:26:54.586670 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:54.586684 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:54.586727 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:26:54.588553 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:54.588580 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:54.589094 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:54.589544 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:54.591179 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:54.592634 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:54.593982 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:54.594716 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:54.594945 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:54.600274 (+ 5329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:54.600283 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:54.600299 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:54.600343 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:54.603919 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:54.604870 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:54.604878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:54.604885 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:54.605091 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:54.605587 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:54.605596 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:54.605599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:54.606289 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:54.606555 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:54.607090 (+ 535us) server_negotiation.cc:299] Negotiation successful
0923 17:26:54.607693 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":397,"threads_started":1}
I20260923 17:26:54.610287 17320 raft_consensus.cc:348] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c142334c6b0e4ef994356563ae4f72dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260923 17:26:54.610687 17320 raft_consensus.cc:374] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:54.610826 17320 raft_consensus.cc:729] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c142334c6b0e4ef994356563ae4f72dd, State: Initialized, Role: FOLLOWER
I20260923 17:26:54.611476 17320 consensus_queue.cc:260] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c142334c6b0e4ef994356563ae4f72dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260923 17:26:54.611755 17320 raft_consensus.cc:388] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:26:54.611894 17320 raft_consensus.cc:482] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:26:54.612061 17320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:54.613307 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:26:54.615195 17320 raft_consensus.cc:504] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c142334c6b0e4ef994356563ae4f72dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260923 17:26:54.615825 17320 leader_election.cc:302] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c142334c6b0e4ef994356563ae4f72dd; no voters:
I20260923 17:26:54.616623 17320 leader_election.cc:288] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:26:54.616755 17325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:54.617617 17325 raft_consensus.cc:686] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [term 1 LEADER]: Becoming Leader. State: Replica: c142334c6b0e4ef994356563ae4f72dd, State: Running, Role: LEADER
I20260923 17:26:54.618461 17325 consensus_queue.cc:237] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c142334c6b0e4ef994356563ae4f72dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260923 17:26:54.620738 17320 sys_catalog.cc:564] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:26:54.621574 17327 sys_catalog.cc:455] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c142334c6b0e4ef994356563ae4f72dd. Latest consensus state: current_term: 1 leader_uuid: "c142334c6b0e4ef994356563ae4f72dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c142334c6b0e4ef994356563ae4f72dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } } }
I20260923 17:26:54.622195 17327 sys_catalog.cc:458] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:54.623198 17329 sys_catalog.cc:455] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c142334c6b0e4ef994356563ae4f72dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c142334c6b0e4ef994356563ae4f72dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } } }
I20260923 17:26:54.624835 17329 sys_catalog.cc:458] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:54.626888 17334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:26:54.633626 17334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:26:54.641549 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:54.621097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50741 (local address 127.0.0.1:38569)
0923 17:26:54.621334 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:54.621341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:54.622341 (+ 1000us) server_negotiation.cc:407] Connection header received
0923 17:26:54.622618 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:54.622625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:54.622702 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:54.622823 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:54.624633 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:54.625926 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:54.628147 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:54.628873 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:54.629040 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:54.632868 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:54.632877 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:54.632883 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:54.632960 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:54.637212 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:54.638192 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:54.638197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:54.638200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:54.638366 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:54.638872 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:54.638877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:54.638880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:54.639383 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:54.639641 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:54.640912 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0923 17:26:54.641151 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:26:54.644855 17334 catalog_manager.cc:1347] Generated new cluster ID: 75e03790659a429a969898e6b7072d1b
I20260923 17:26:54.645067 17334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:26:54.657923 17334 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:26:54.660065 17334 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:26:54.685182 17334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c142334c6b0e4ef994356563ae4f72dd: Generated new TSK 0
I20260923 17:26:54.686458 17334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:26:54.701583 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17253
I20260923 17:26:54.702805 17315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:26:54.799526 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 17253
2026-09-23T17:26:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2077 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:26:54 dist-test-slave-j0m4 krb5kdc[17352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:26:54 dist-test-slave-j0m4 krb5kdc[17352](info): set up 2 sockets
Sep 23 17:26:54 dist-test-slave-j0m4 krb5kdc[17352](info): commencing operation
krb5kdc: starting...
W20260923 17:26:55.923691 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:26:56 dist-test-slave-j0m4 krb5kdc[17352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:26:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:26:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:26:56.207429 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:56.564908 17365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:56.565229 17365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:56.565375 17365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:56.573031 17365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:26:56.573160 17365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:56.573279 17365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:56.573400 17365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:26:56.573454 17365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:26:56.583906 17365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:56.585506 17365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:56.586877 17365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:56.599428 17370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:56.599402 17371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:56.599457 17373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:56.600183 17365 server_base.cc:1034] running on GCE node
I20260923 17:26:56.601028 17365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:56.602890 17365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:56.604120 17365 hybrid_clock.cc:648] HybridClock initialized: now 1790184416604079 us; error 72 us; skew 500 ppm
I20260923 17:26:56.604584 17365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:56 dist-test-slave-j0m4 krb5kdc[17352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:56.608093 17365 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:26:56.609771 17365 webserver.cc:460] Webserver started at http://127.0.0.1:43983/ using document root <none> and password file <none>
I20260923 17:26:56.610472 17365 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:56.610586 17365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:56.610859 17365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:56.614205 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "b0def4480d854ecc931f9e62a0febd50"
format_stamp: "Formatted at 2026-09-23 17:26:56 on dist-test-slave-j0m4"
server_key: "65bd972566582bb3503f969524cf74a2"
server_key_iv: "c70d62a4d6aa9280887b4633e6bd3469"
server_key_version: "encryptionkey@0"
I20260923 17:26:56.615034 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "b0def4480d854ecc931f9e62a0febd50"
format_stamp: "Formatted at 2026-09-23 17:26:56 on dist-test-slave-j0m4"
server_key: "65bd972566582bb3503f969524cf74a2"
server_key_iv: "c70d62a4d6aa9280887b4633e6bd3469"
server_key_version: "encryptionkey@0"
I20260923 17:26:56.620850 17365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:26:56.624863 17380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:56.626353 17365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:26:56.626565 17365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "b0def4480d854ecc931f9e62a0febd50"
format_stamp: "Formatted at 2026-09-23 17:26:56 on dist-test-slave-j0m4"
server_key: "65bd972566582bb3503f969524cf74a2"
server_key_iv: "c70d62a4d6aa9280887b4633e6bd3469"
server_key_version: "encryptionkey@0"
I20260923 17:26:56.626791 17365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:56.644449 17365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:56.649616 17365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:56.649961 17365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:56.670485 17365 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35429
I20260923 17:26:56.670522 17431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35429 every 8 connection(s)
I20260923 17:26:56.672024 17365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:26:56.675302 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 17365
I20260923 17:26:56.675508 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:26:56.676055 28963 external_mini_cluster.cc:1428] Setting key 4f97bd0f4c7201997a15bcbf0ee55e88
I20260923 17:26:56.676771 17432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:56.692605 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:56.678235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44085 (local address 127.0.0.1:35429)
0923 17:26:56.679260 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:56.679273 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:56.679320 (+ 47us) server_negotiation.cc:407] Connection header received
0923 17:26:56.681925 (+ 2605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:56.681953 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:56.682479 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:56.682945 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:56.683881 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:56.685340 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:56.687475 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:56.688253 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:56.688426 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:56.689956 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:56.689970 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:56.689987 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:56.690609 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:56.690695 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:56.690727 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:56.690824 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:56.691095 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 17:26:56.691538 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":449,"threads_started":1}
I20260923 17:26:56.692574 17432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50: Bootstrap starting.
I20260923 17:26:56.696249 17432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:56.696816 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:26:56.697662 17432 log.cc:824] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:56.701452 17432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50: No bootstrap required, opened a new log
I20260923 17:26:56.709183 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:56.702099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41285 (local address 127.0.0.1:35429)
0923 17:26:56.702306 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:56.702314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:56.702772 (+ 458us) server_negotiation.cc:407] Connection header received
0923 17:26:56.702931 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:56.702936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:56.703021 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:56.703185 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:56.703737 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:56.704955 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:56.706317 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:56.707090 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:56.707250 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:56.708357 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:56.708365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:56.708370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:56.708394 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:56.708453 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:56.708475 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:56.708574 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:56.708708 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 17:26:56.708806 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:26:56.711282 17432 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0def4480d854ecc931f9e62a0febd50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } }
I20260923 17:26:56.711702 17432 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:56.711829 17432 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0def4480d854ecc931f9e62a0febd50, State: Initialized, Role: FOLLOWER
I20260923 17:26:56.712445 17432 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0def4480d854ecc931f9e62a0febd50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } }
I20260923 17:26:56.712688 17432 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:26:56.712823 17432 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:26:56.712996 17432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:56.715354 17432 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0def4480d854ecc931f9e62a0febd50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } }
I20260923 17:26:56.715902 17432 leader_election.cc:302] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0def4480d854ecc931f9e62a0febd50; no voters:
I20260923 17:26:56.716620 17432 leader_election.cc:288] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:26:56.716787 17442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:56.718037 17442 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [term 1 LEADER]: Becoming Leader. State: Replica: b0def4480d854ecc931f9e62a0febd50, State: Running, Role: LEADER
I20260923 17:26:56.718676 17442 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0def4480d854ecc931f9e62a0febd50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } }
I20260923 17:26:56.719818 17432 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:26:56.722013 17443 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0def4480d854ecc931f9e62a0febd50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0def4480d854ecc931f9e62a0febd50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } } }
I20260923 17:26:56.721985 17444 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0def4480d854ecc931f9e62a0febd50. Latest consensus state: current_term: 1 leader_uuid: "b0def4480d854ecc931f9e62a0febd50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0def4480d854ecc931f9e62a0febd50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } } }
I20260923 17:26:56.722515 17443 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:56.722520 17444 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:56.724849 17448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:26:56.731901 17448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:26:56.741616 17448 catalog_manager.cc:1347] Generated new cluster ID: 3ea1edb58b1748878268bf8db2e028a7
I20260923 17:26:56.741760 17448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:26:56.754104 17448 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:26:56.756464 17448 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:26:56.769835 17448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0def4480d854ecc931f9e62a0febd50: Generated new TSK 0
I20260923 17:26:56.771162 17448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:26:56.784825 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17365
W20260923 17:26:56.786017 17427 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44085
W20260923 17:26:56.786847 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17365: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:26:56.786983 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 17365
2026-09-23T17:26:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1986 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:26:56 dist-test-slave-j0m4 krb5kdc[17463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:26:56 dist-test-slave-j0m4 krb5kdc[17463](info): set up 2 sockets
Sep 23 17:26:56 dist-test-slave-j0m4 krb5kdc[17463](info): commencing operation
krb5kdc: starting...
W20260923 17:26:57.919711 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:26:58 dist-test-slave-j0m4 krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:26:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:26:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:26:58.199788 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:58.542315 17476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:58.542634 17476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:58.542804 17476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:58.550361 17476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:26:58.550493 17476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:58.550588 17476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:58.550695 17476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:26:58.550776 17476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:26:58.561474 17476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:26:58.562894 17476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:58.564291 17476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:58.577354 17484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:58.577327 17482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:58.578104 17481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:58.578212 17476 server_base.cc:1034] running on GCE node
I20260923 17:26:58.579219 17476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:58.580818 17476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:58.582070 17476 hybrid_clock.cc:648] HybridClock initialized: now 1790184418582018 us; error 48 us; skew 500 ppm
I20260923 17:26:58.582581 17476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:26:58 dist-test-slave-j0m4 krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:58.590590 17476 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:26:58.592594 17476 webserver.cc:460] Webserver started at http://127.0.0.1:36043/ using document root <none> and password file <none>
I20260923 17:26:58.593400 17476 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:58.593564 17476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:58.593912 17476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:26:58.597230 17476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "3d5188b43f9d4c64a54b0dd4c91eac91"
format_stamp: "Formatted at 2026-09-23 17:26:58 on dist-test-slave-j0m4"
server_key: "876cd53fc73b7eb220653c09d990c7c6"
server_key_iv: "af00d19214239ccc04bb7d5163665b46"
server_key_version: "encryptionkey@0"
I20260923 17:26:58.598079 17476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "3d5188b43f9d4c64a54b0dd4c91eac91"
format_stamp: "Formatted at 2026-09-23 17:26:58 on dist-test-slave-j0m4"
server_key: "876cd53fc73b7eb220653c09d990c7c6"
server_key_iv: "af00d19214239ccc04bb7d5163665b46"
server_key_version: "encryptionkey@0"
I20260923 17:26:58.604000 17476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260923 17:26:58.608003 17491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:58.609555 17476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:26:58.609804 17476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "3d5188b43f9d4c64a54b0dd4c91eac91"
format_stamp: "Formatted at 2026-09-23 17:26:58 on dist-test-slave-j0m4"
server_key: "876cd53fc73b7eb220653c09d990c7c6"
server_key_iv: "af00d19214239ccc04bb7d5163665b46"
server_key_version: "encryptionkey@0"
I20260923 17:26:58.610035 17476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:58.616468 17476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:58.621454 17476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:58.621865 17476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:58.642906 17476 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41907
I20260923 17:26:58.642956 17542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41907 every 8 connection(s)
I20260923 17:26:58.644520 17476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:26:58.647039 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 17476
I20260923 17:26:58.647287 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:26:58.647850 28963 external_mini_cluster.cc:1428] Setting key ad46ff15ed1154980a4f1623f3baedec
I20260923 17:26:58.649677 17543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:58.656840 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:58.649947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35095 (local address 127.0.0.1:41907)
0923 17:26:58.650850 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:58.650863 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:58.650915 (+ 52us) server_negotiation.cc:407] Connection header received
0923 17:26:58.652672 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:58.652698 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:58.653110 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:58.653492 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:58.654830 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:58.654837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:58.654853 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:58.654877 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:58.654971 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:58.654994 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:58.655332 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:58.655508 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 17:26:58.655760 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":424,"threads_started":1}
I20260923 17:26:58.661283 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:26:58.663308 17543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91: Bootstrap starting.
I20260923 17:26:58.667570 17543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:58.669209 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:58.666206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58043 (local address 127.0.0.1:41907)
0923 17:26:58.666432 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:58.666441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:58.666885 (+ 444us) server_negotiation.cc:407] Connection header received
0923 17:26:58.667049 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:58.667054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:58.667145 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:58.667269 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:58.668418 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:58.668424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:58.668428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:26:58.668449 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:58.668503 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:26:58.668523 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:58.668656 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:58.668823 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 17:26:58.668894 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:26:58.669173 17543 log.cc:824] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:58.672516 17543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91: No bootstrap required, opened a new log
I20260923 17:26:58.680609 17543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5188b43f9d4c64a54b0dd4c91eac91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } }
I20260923 17:26:58.681068 17543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:58.681205 17543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5188b43f9d4c64a54b0dd4c91eac91, State: Initialized, Role: FOLLOWER
I20260923 17:26:58.681854 17543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5188b43f9d4c64a54b0dd4c91eac91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } }
I20260923 17:26:58.682099 17543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:26:58.682278 17543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:26:58.682459 17543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:58.685091 17543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5188b43f9d4c64a54b0dd4c91eac91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } }
I20260923 17:26:58.685731 17543 leader_election.cc:302] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d5188b43f9d4c64a54b0dd4c91eac91; no voters:
I20260923 17:26:58.686443 17543 leader_election.cc:288] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:26:58.686636 17553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:58.687363 17553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [term 1 LEADER]: Becoming Leader. State: Replica: 3d5188b43f9d4c64a54b0dd4c91eac91, State: Running, Role: LEADER
I20260923 17:26:58.687939 17553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5188b43f9d4c64a54b0dd4c91eac91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } }
I20260923 17:26:58.688750 17543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:26:58.690896 17554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d5188b43f9d4c64a54b0dd4c91eac91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5188b43f9d4c64a54b0dd4c91eac91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } } }
I20260923 17:26:58.691493 17554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:58.691655 17555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d5188b43f9d4c64a54b0dd4c91eac91. Latest consensus state: current_term: 1 leader_uuid: "3d5188b43f9d4c64a54b0dd4c91eac91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5188b43f9d4c64a54b0dd4c91eac91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } } }
I20260923 17:26:58.692050 17555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:58.694581 17558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:26:58.700280 17558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:26:58.709932 17558 catalog_manager.cc:1347] Generated new cluster ID: a6ec0908e72141cb8cbbed41cb166cb5
I20260923 17:26:58.710075 17558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:26:58.735172 17558 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:26:58.735692 17558 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:26:58.744500 17558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d5188b43f9d4c64a54b0dd4c91eac91: Generated new TSK 0
I20260923 17:26:58.745707 17558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:26:58.762195 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17476
W20260923 17:26:58.763680 17538 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35095
W20260923 17:26:58.764420 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17476: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:26:58.764581 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 17476
2026-09-23T17:26:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1977 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:26:58 dist-test-slave-j0m4 krb5kdc[17574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:26:58 dist-test-slave-j0m4 krb5kdc[17574](info): set up 2 sockets
Sep 23 17:26:58 dist-test-slave-j0m4 krb5kdc[17574](info): commencing operation
krb5kdc: starting...
W20260923 17:26:59.911720 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:27:00 dist-test-slave-j0m4 krb5kdc[17574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:27:00.198738 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35563
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:00.555652 17587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:00.556051 17587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:00.556222 17587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:00.564141 17587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:00.564337 17587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:00.564463 17587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:00.564551 17587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:00.564634 17587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:00.575096 17587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:00.576628 17587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:00.578053 17587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:00.591225 17592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:00.591162 17593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:00.591189 17595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:00.592113 17587 server_base.cc:1034] running on GCE node
I20260923 17:27:00.592981 17587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:00.594590 17587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:00.595854 17587 hybrid_clock.cc:648] HybridClock initialized: now 1790184420595782 us; error 64 us; skew 500 ppm
I20260923 17:27:00.596266 17587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:00 dist-test-slave-j0m4 krb5kdc[17574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:00.599786 17587 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:27:00.601527 17587 webserver.cc:460] Webserver started at http://127.0.0.1:33209/ using document root <none> and password file <none>
I20260923 17:27:00.602237 17587 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:00.602404 17587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:00.602730 17587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:00.605876 17587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "77db93bb5eed4b6ab73f5fd3efdf2602"
format_stamp: "Formatted at 2026-09-23 17:27:00 on dist-test-slave-j0m4"
server_key: "384a59659d82728e54dba5600f8189ce"
server_key_iv: "73a02c67c9899f592d106b368e58353a"
server_key_version: "encryptionkey@0"
I20260923 17:27:00.606709 17587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "77db93bb5eed4b6ab73f5fd3efdf2602"
format_stamp: "Formatted at 2026-09-23 17:27:00 on dist-test-slave-j0m4"
server_key: "384a59659d82728e54dba5600f8189ce"
server_key_iv: "73a02c67c9899f592d106b368e58353a"
server_key_version: "encryptionkey@0"
I20260923 17:27:00.612867 17587 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 17:27:00.616917 17602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:00.618685 17587 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:27:00.618901 17587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "77db93bb5eed4b6ab73f5fd3efdf2602"
format_stamp: "Formatted at 2026-09-23 17:27:00 on dist-test-slave-j0m4"
server_key: "384a59659d82728e54dba5600f8189ce"
server_key_iv: "73a02c67c9899f592d106b368e58353a"
server_key_version: "encryptionkey@0"
I20260923 17:27:00.619146 17587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:00.669631 17587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:00.674289 17587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:00.674614 17587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:00.695946 17587 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35563
I20260923 17:27:00.695996 17653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35563 every 8 connection(s)
I20260923 17:27:00.697453 17587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:00.700248 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 17587
I20260923 17:27:00.700435 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:27:00.700980 28963 external_mini_cluster.cc:1428] Setting key 1260734fb7a858a47ef18f4a25aba3e4
I20260923 17:27:00.701838 17654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:27:00 dist-test-slave-j0m4 krb5kdc[17574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:27:00.717451 17654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602: Bootstrap starting.
I20260923 17:27:00.721457 17654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:00.722674 17654 log.cc:824] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:00.726230 17654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602: No bootstrap required, opened a new log
I20260923 17:27:00.726586 17657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:00.702587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:39459 (local address 127.0.0.1:35563)
0923 17:27:00.703426 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:00.703438 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:00.703478 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:27:00.705214 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:00.705236 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:00.705791 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:00.706151 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:00.708222 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:00.709693 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:00.711574 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:00.712476 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:00.712634 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:00.717986 (+ 5352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:00.717995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:00.718015 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:00.718062 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:00.721466 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:00.722542 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:00.722549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:00.722556 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:00.722739 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:00.723453 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:00.723459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:00.723466 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:00.724107 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:00.724499 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:00.725070 (+ 571us) server_negotiation.cc:299] Negotiation successful
0923 17:27:00.725470 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":377,"threads_started":1}
I20260923 17:27:00.731483 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:00.735426 17654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77db93bb5eed4b6ab73f5fd3efdf2602" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35563 } }
I20260923 17:27:00.735869 17654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:00.735983 17654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77db93bb5eed4b6ab73f5fd3efdf2602, State: Initialized, Role: FOLLOWER
I20260923 17:27:00.736693 17654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77db93bb5eed4b6ab73f5fd3efdf2602" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35563 } }
I20260923 17:27:00.736953 17654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:00.737098 17654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:00.737313 17654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:00.739724 17654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77db93bb5eed4b6ab73f5fd3efdf2602" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35563 } }
I20260923 17:27:00.740288 17654 leader_election.cc:302] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77db93bb5eed4b6ab73f5fd3efdf2602; no voters:
I20260923 17:27:00.740962 17654 leader_election.cc:288] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:00.741119 17664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:00.743212 17654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:00.743938 17664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [term 1 LEADER]: Becoming Leader. State: Replica: 77db93bb5eed4b6ab73f5fd3efdf2602, State: Running, Role: LEADER
I20260923 17:27:00.744638 17664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77db93bb5eed4b6ab73f5fd3efdf2602" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35563 } }
I20260923 17:27:00.747944 17665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77db93bb5eed4b6ab73f5fd3efdf2602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77db93bb5eed4b6ab73f5fd3efdf2602" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35563 } } }
I20260923 17:27:00.748523 17665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:00.747637 17666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77db93bb5eed4b6ab73f5fd3efdf2602. Latest consensus state: current_term: 1 leader_uuid: "77db93bb5eed4b6ab73f5fd3efdf2602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77db93bb5eed4b6ab73f5fd3efdf2602" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35563 } } }
I20260923 17:27:00.757802 17657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:00.736208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:39511 (local address 127.0.0.1:35563)
0923 17:27:00.736445 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:00.736455 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:00.737039 (+ 584us) server_negotiation.cc:407] Connection header received
0923 17:27:00.737262 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:00.737267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:00.737353 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:00.737461 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:00.738931 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:00.739799 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:00.741391 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:00.742328 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:00.742464 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:00.746408 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:00.746416 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:00.746425 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:00.746474 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:00.751557 (+ 5083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:00.753253 (+ 1696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:00.753261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:00.753264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:00.753530 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:00.754472 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:00.754480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:00.754486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:00.755019 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:00.755451 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:00.756983 (+ 1532us) server_negotiation.cc:299] Negotiation successful
0923 17:27:00.757269 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:27:00.757970 17666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:00.761058 17674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:00.765949 17674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:00.775856 17674 catalog_manager.cc:1347] Generated new cluster ID: 1f1db3af4f274794a31d45ef95e33e68
I20260923 17:27:00.776057 17674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:00.793395 17674 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:00.795574 17674 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:00.805833 17674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77db93bb5eed4b6ab73f5fd3efdf2602: Generated new TSK 0
I20260923 17:27:00.807204 17674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:00.819691 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17587
I20260923 17:27:00.821122 17649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:27:00.917248 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 17587
W20260923 17:27:00.934975 17572 connection.cc:535] client connection to 127.0.0.1:35563 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35563): Connection reset by peer (error 104)
2026-09-23T17:27:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2155 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:01 dist-test-slave-j0m4 krb5kdc[17687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:01 dist-test-slave-j0m4 krb5kdc[17687](info): set up 2 sockets
Sep 23 17:27:01 dist-test-slave-j0m4 krb5kdc[17687](info): commencing operation
krb5kdc: starting...
W20260923 17:27:02.092356 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:27:02 dist-test-slave-j0m4 krb5kdc[17687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:27:02.379136 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:02.739595 17700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:02.739982 17700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:02.740156 17700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:02.748057 17700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:02.748255 17700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:02.748384 17700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:02.748497 17700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:02.748580 17700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:02.759225 17700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:02.760711 17700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:02.762115 17700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:02.775348 17706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:02.775437 17708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:02.775458 17705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:02.776569 17700 server_base.cc:1034] running on GCE node
I20260923 17:27:02.777482 17700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:02.779071 17700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:02.780351 17700 hybrid_clock.cc:648] HybridClock initialized: now 1790184422780309 us; error 43 us; skew 500 ppm
I20260923 17:27:02.780766 17700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:02 dist-test-slave-j0m4 krb5kdc[17687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:02.784139 17700 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:27:02.785920 17700 webserver.cc:460] Webserver started at http://127.0.0.1:44415/ using document root <none> and password file <none>
I20260923 17:27:02.786729 17700 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:02.786880 17700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:02.787276 17700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:02.790467 17700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "094a4e28b1fd431ba7d4bbb546741c5e"
format_stamp: "Formatted at 2026-09-23 17:27:02 on dist-test-slave-j0m4"
server_key: "c76090d9e8f8f6ae670010ea7cf4a088"
server_key_iv: "467ade7a4267ed875e6122924a183470"
server_key_version: "encryptionkey@0"
I20260923 17:27:02.791437 17700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "094a4e28b1fd431ba7d4bbb546741c5e"
format_stamp: "Formatted at 2026-09-23 17:27:02 on dist-test-slave-j0m4"
server_key: "c76090d9e8f8f6ae670010ea7cf4a088"
server_key_iv: "467ade7a4267ed875e6122924a183470"
server_key_version: "encryptionkey@0"
I20260923 17:27:02.797333 17700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:27:02.801611 17715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:02.803692 17700 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 17:27:02.803938 17700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "094a4e28b1fd431ba7d4bbb546741c5e"
format_stamp: "Formatted at 2026-09-23 17:27:02 on dist-test-slave-j0m4"
server_key: "c76090d9e8f8f6ae670010ea7cf4a088"
server_key_iv: "467ade7a4267ed875e6122924a183470"
server_key_version: "encryptionkey@0"
I20260923 17:27:02.804184 17700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:02.819639 17700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:02.824383 17700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:02.824692 17700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:02.844966 17700 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44191
I20260923 17:27:02.845000 17766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44191 every 8 connection(s)
I20260923 17:27:02.846458 17700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:02.848035 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 17700
I20260923 17:27:02.848218 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:27:02.848856 28963 external_mini_cluster.cc:1428] Setting key ed4abaf3c2d2dc844d2a3ac056de8aa2
I20260923 17:27:02.851660 17767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:02.863405 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:02.850680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:47471 (local address 127.0.0.1:44191)
0923 17:27:02.851530 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:02.851542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:02.851584 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:27:02.853362 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:02.853385 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:02.853781 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:02.854140 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:02.854869 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:02.856386 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:02.858346 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:02.859062 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:02.859308 (+ 246us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:02.860706 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:02.860714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:02.860735 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:02.861329 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:02.861419 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:02.861449 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:02.861576 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:02.862016 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 17:27:02.862257 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":377,"threads_started":1}
I20260923 17:27:02.864912 17767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e: Bootstrap starting.
I20260923 17:27:02.867808 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:02.868660 17767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:02.869993 17767 log.cc:824] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:02.873556 17767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e: No bootstrap required, opened a new log
I20260923 17:27:02.880710 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:02.873247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:41211 (local address 127.0.0.1:44191)
0923 17:27:02.873481 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:02.873490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:02.873944 (+ 454us) server_negotiation.cc:407] Connection header received
0923 17:27:02.874153 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:02.874157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:02.874238 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:02.874325 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:02.875048 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:02.875918 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:02.877901 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:02.878384 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:02.878495 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:02.879876 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:02.879881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:02.879885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:02.879907 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:02.879953 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:02.879970 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:02.880058 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:02.880281 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 17:27:02.880329 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:27:02.883579 17767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094a4e28b1fd431ba7d4bbb546741c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } }
I20260923 17:27:02.884055 17767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:02.884174 17767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094a4e28b1fd431ba7d4bbb546741c5e, State: Initialized, Role: FOLLOWER
I20260923 17:27:02.884793 17767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094a4e28b1fd431ba7d4bbb546741c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } }
I20260923 17:27:02.885051 17767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:02.885208 17767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:02.885396 17767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:02.888072 17767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094a4e28b1fd431ba7d4bbb546741c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } }
I20260923 17:27:02.888630 17767 leader_election.cc:302] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 094a4e28b1fd431ba7d4bbb546741c5e; no voters:
I20260923 17:27:02.889286 17767 leader_election.cc:288] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:02.889442 17777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:02.890162 17777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [term 1 LEADER]: Becoming Leader. State: Replica: 094a4e28b1fd431ba7d4bbb546741c5e, State: Running, Role: LEADER
I20260923 17:27:02.890769 17777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094a4e28b1fd431ba7d4bbb546741c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } }
I20260923 17:27:02.891572 17767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:02.894183 17779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094a4e28b1fd431ba7d4bbb546741c5e. Latest consensus state: current_term: 1 leader_uuid: "094a4e28b1fd431ba7d4bbb546741c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094a4e28b1fd431ba7d4bbb546741c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } } }
I20260923 17:27:02.894631 17779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:02.894169 17778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "094a4e28b1fd431ba7d4bbb546741c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094a4e28b1fd431ba7d4bbb546741c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } } }
I20260923 17:27:02.895565 17783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:02.900039 17778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:02.906428 17783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:02.915720 17783 catalog_manager.cc:1347] Generated new cluster ID: e722e047c49e4a97a8e07eb24dd5ba64
I20260923 17:27:02.915877 17783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:02.935479 17783 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:02.936867 17783 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:02.945498 17783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 094a4e28b1fd431ba7d4bbb546741c5e: Generated new TSK 0
I20260923 17:27:02.946477 17783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:02.962246 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17700
W20260923 17:27:02.963948 17762 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.197:47471
W20260923 17:27:02.964964 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17700: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:27:02.965161 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 17700
W20260923 17:27:02.980963 17685 connection.cc:535] client connection to 127.0.0.1:44191 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44191): Connection reset by peer (error 104)
2026-09-23T17:27:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2045 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:03 dist-test-slave-j0m4 krb5kdc[17798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:03 dist-test-slave-j0m4 krb5kdc[17798](info): set up 2 sockets
Sep 23 17:27:03 dist-test-slave-j0m4 krb5kdc[17798](info): commencing operation
krb5kdc: starting...
W20260923 17:27:04.091858 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:27:04 dist-test-slave-j0m4 krb5kdc[17798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:27:04.382988 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:04.725875 17811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:04.726192 17811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:04.726359 17811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:04.734197 17811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:04.734395 17811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:04.734488 17811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:04.734593 17811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:04.734669 17811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:04.745214 17811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:04.746683 17811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:04.748090 17811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:04.761160 17816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:04.761116 17817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:04.761075 17819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:04.761870 17811 server_base.cc:1034] running on GCE node
I20260923 17:27:04.762828 17811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:04.764532 17811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:04.765750 17811 hybrid_clock.cc:648] HybridClock initialized: now 1790184424765705 us; error 45 us; skew 500 ppm
I20260923 17:27:04.766175 17811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:04 dist-test-slave-j0m4 krb5kdc[17798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:04.774482 17811 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:27:04.776294 17811 webserver.cc:460] Webserver started at http://127.0.0.1:34203/ using document root <none> and password file <none>
I20260923 17:27:04.777060 17811 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:04.777172 17811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:04.777509 17811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:04.780705 17811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "c52b8f400ab545ae9d1de11ac75bf6c4"
format_stamp: "Formatted at 2026-09-23 17:27:04 on dist-test-slave-j0m4"
server_key: "ddfa550bfee76f61a922f2007e6f2794"
server_key_iv: "9fc168bba177124f5bacfcf4822df569"
server_key_version: "encryptionkey@0"
I20260923 17:27:04.781526 17811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "c52b8f400ab545ae9d1de11ac75bf6c4"
format_stamp: "Formatted at 2026-09-23 17:27:04 on dist-test-slave-j0m4"
server_key: "ddfa550bfee76f61a922f2007e6f2794"
server_key_iv: "9fc168bba177124f5bacfcf4822df569"
server_key_version: "encryptionkey@0"
I20260923 17:27:04.787317 17811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:27:04.791280 17826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:04.792999 17811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:27:04.793227 17811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "c52b8f400ab545ae9d1de11ac75bf6c4"
format_stamp: "Formatted at 2026-09-23 17:27:04 on dist-test-slave-j0m4"
server_key: "ddfa550bfee76f61a922f2007e6f2794"
server_key_iv: "9fc168bba177124f5bacfcf4822df569"
server_key_version: "encryptionkey@0"
I20260923 17:27:04.793453 17811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:04.800398 17811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:04.805560 17811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:04.805898 17811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:04.825678 17811 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45457
I20260923 17:27:04.825721 17877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45457 every 8 connection(s)
I20260923 17:27:04.827234 17811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:04.829957 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 17811
I20260923 17:27:04.830108 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:27:04.830587 28963 external_mini_cluster.cc:1428] Setting key f7d07f21d4cd454b8308d82a54450dbe
I20260923 17:27:04.831768 17878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:04.838577 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:04.832583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:48891 (local address 127.0.0.1:45457)
0923 17:27:04.833403 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:04.833414 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:04.833456 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:27:04.835035 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:04.835056 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:04.835480 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:04.835824 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:04.836925 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:04.836931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:04.836974 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:04.836989 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:04.837075 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:04.837095 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:04.837264 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:04.837382 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 17:27:04.837608 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":365,"threads_started":1}
I20260923 17:27:04.842427 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:04.845371 17878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4: Bootstrap starting.
I20260923 17:27:04.849258 17878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:04.849612 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:04.846853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:33023 (local address 127.0.0.1:45457)
0923 17:27:04.847065 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:04.847073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:04.847604 (+ 531us) server_negotiation.cc:407] Connection header received
0923 17:27:04.847825 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:04.847830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:04.847880 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:04.847991 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:04.848948 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:04.848955 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:04.848960 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:04.848980 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:04.849027 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:04.849043 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:04.849168 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:04.849265 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 17:27:04.849332 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:27:04.850639 17878 log.cc:824] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:04.853657 17878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4: No bootstrap required, opened a new log
I20260923 17:27:04.862170 17878 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52b8f400ab545ae9d1de11ac75bf6c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260923 17:27:04.862704 17878 raft_consensus.cc:374] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:04.862841 17878 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52b8f400ab545ae9d1de11ac75bf6c4, State: Initialized, Role: FOLLOWER
I20260923 17:27:04.863682 17878 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52b8f400ab545ae9d1de11ac75bf6c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260923 17:27:04.863974 17878 raft_consensus.cc:388] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:04.864153 17878 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:04.864383 17878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:04.867264 17878 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52b8f400ab545ae9d1de11ac75bf6c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260923 17:27:04.867875 17878 leader_election.cc:302] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c52b8f400ab545ae9d1de11ac75bf6c4; no voters:
I20260923 17:27:04.868655 17878 leader_election.cc:288] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:04.868832 17888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:04.869768 17888 raft_consensus.cc:686] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [term 1 LEADER]: Becoming Leader. State: Replica: c52b8f400ab545ae9d1de11ac75bf6c4, State: Running, Role: LEADER
I20260923 17:27:04.870486 17888 consensus_queue.cc:237] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52b8f400ab545ae9d1de11ac75bf6c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260923 17:27:04.871506 17878 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:04.874022 17890 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52b8f400ab545ae9d1de11ac75bf6c4. Latest consensus state: current_term: 1 leader_uuid: "c52b8f400ab545ae9d1de11ac75bf6c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52b8f400ab545ae9d1de11ac75bf6c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } } }
I20260923 17:27:04.874707 17890 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:04.875164 17889 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c52b8f400ab545ae9d1de11ac75bf6c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52b8f400ab545ae9d1de11ac75bf6c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } } }
I20260923 17:27:04.875476 17889 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:04.877331 17893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:04.883081 17893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:04.894124 17893 catalog_manager.cc:1347] Generated new cluster ID: 01dab5eff50d420db7b4ba2485c9421b
I20260923 17:27:04.894280 17893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:04.912086 17893 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:04.912599 17893 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:04.921638 17893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c52b8f400ab545ae9d1de11ac75bf6c4: Generated new TSK 0
I20260923 17:27:04.922812 17893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:04.932430 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17811
W20260923 17:27:04.933751 17873 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.197:48891
W20260923 17:27:04.934576 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17811: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:27:04.934731 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 17811
2026-09-23T17:27:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1970 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:05 dist-test-slave-j0m4 krb5kdc[17909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:05 dist-test-slave-j0m4 krb5kdc[17909](info): set up 2 sockets
Sep 23 17:27:05 dist-test-slave-j0m4 krb5kdc[17909](info): commencing operation
krb5kdc: starting...
W20260923 17:27:06.108044 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:27:06 dist-test-slave-j0m4 krb5kdc[17909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:27:06.386644 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:06.742071 17922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:06.742491 17922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:06.742666 17922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:06.750185 17922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:06.750378 17922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:06.750499 17922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:06.750609 17922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:06.750698 17922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:06.760941 17922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:06.762430 17922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:06.764014 17922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:06.776728 17927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:06.776818 17928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:06.776995 17930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:06.778049 17922 server_base.cc:1034] running on GCE node
I20260923 17:27:06.779559 17922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:06.781243 17922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:06.782532 17922 hybrid_clock.cc:648] HybridClock initialized: now 1790184426782480 us; error 41 us; skew 500 ppm
I20260923 17:27:06.782961 17922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:06 dist-test-slave-j0m4 krb5kdc[17909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:06.786350 17922 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:27:06.788089 17922 webserver.cc:460] Webserver started at http://127.0.0.1:36555/ using document root <none> and password file <none>
I20260923 17:27:06.788823 17922 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:06.788980 17922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:06.789299 17922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:06.792644 17922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "fbf330722b574446b1f27be522d840cf"
format_stamp: "Formatted at 2026-09-23 17:27:06 on dist-test-slave-j0m4"
server_key: "32c1bde59e781226f780e3238d701aa5"
server_key_iv: "af467040220e7d505ec1fbcc00588dda"
server_key_version: "encryptionkey@0"
I20260923 17:27:06.793535 17922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "fbf330722b574446b1f27be522d840cf"
format_stamp: "Formatted at 2026-09-23 17:27:06 on dist-test-slave-j0m4"
server_key: "32c1bde59e781226f780e3238d701aa5"
server_key_iv: "af467040220e7d505ec1fbcc00588dda"
server_key_version: "encryptionkey@0"
I20260923 17:27:06.799621 17922 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 17:27:06.803706 17937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:06.805519 17922 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:27:06.805753 17922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "fbf330722b574446b1f27be522d840cf"
format_stamp: "Formatted at 2026-09-23 17:27:06 on dist-test-slave-j0m4"
server_key: "32c1bde59e781226f780e3238d701aa5"
server_key_iv: "af467040220e7d505ec1fbcc00588dda"
server_key_version: "encryptionkey@0"
I20260923 17:27:06.805994 17922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:06.821817 17922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:06.829128 17922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:06.829470 17922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:06.852250 17922 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41213
I20260923 17:27:06.852283 17988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41213 every 8 connection(s)
I20260923 17:27:06.853942 17922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:06.854560 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 17922
I20260923 17:27:06.854776 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:27:06.855515 28963 external_mini_cluster.cc:1428] Setting key 18eb97cfb452380cddaac909a75a308f
I20260923 17:27:06.859519 17989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:27:06 dist-test-slave-j0m4 krb5kdc[17909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:27:06.873901 17989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf: Bootstrap starting.
I20260923 17:27:06.878710 17989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:06.880288 17989 log.cc:824] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:06.883960 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:06.857603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:55541 (local address 127.0.0.1:41213)
0923 17:27:06.858390 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:06.858406 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:06.858466 (+ 60us) server_negotiation.cc:407] Connection header received
0923 17:27:06.861005 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:06.861034 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:06.861576 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:06.862069 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:06.863787 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:06.865974 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:06.867683 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:06.868802 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:06.869008 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:06.873847 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:06.873861 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:06.873899 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:06.873954 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:06.878112 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:06.879435 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:06.879445 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:06.879453 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:06.879687 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:06.880315 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:06.880321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:06.880323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:06.881149 (+ 826us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:06.881417 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:06.882152 (+ 735us) server_negotiation.cc:299] Negotiation successful
0923 17:27:06.882507 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":307,"threads_started":1}
I20260923 17:27:06.884271 17989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf: No bootstrap required, opened a new log
I20260923 17:27:06.888746 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:06.893208 17989 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf330722b574446b1f27be522d840cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260923 17:27:06.893711 17989 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:06.893868 17989 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf330722b574446b1f27be522d840cf, State: Initialized, Role: FOLLOWER
I20260923 17:27:06.894707 17989 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf330722b574446b1f27be522d840cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260923 17:27:06.895035 17989 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:06.895268 17989 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:06.895511 17989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:06.898787 17989 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf330722b574446b1f27be522d840cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260923 17:27:06.899567 17989 leader_election.cc:302] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbf330722b574446b1f27be522d840cf; no voters:
I20260923 17:27:06.900507 17989 leader_election.cc:288] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:06.900704 17999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:06.902199 17999 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [term 1 LEADER]: Becoming Leader. State: Replica: fbf330722b574446b1f27be522d840cf, State: Running, Role: LEADER
I20260923 17:27:06.903354 17989 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:06.903326 17999 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf330722b574446b1f27be522d840cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260923 17:27:06.907593 18000 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbf330722b574446b1f27be522d840cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf330722b574446b1f27be522d840cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } } }
I20260923 17:27:06.908128 18000 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:06.912580 18001 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbf330722b574446b1f27be522d840cf. Latest consensus state: current_term: 1 leader_uuid: "fbf330722b574446b1f27be522d840cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf330722b574446b1f27be522d840cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } } }
I20260923 17:27:06.913118 18001 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:06.913800 18009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:06.915431 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:06.894053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:49709 (local address 127.0.0.1:41213)
0923 17:27:06.894347 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:06.894358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:06.894997 (+ 639us) server_negotiation.cc:407] Connection header received
0923 17:27:06.895104 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:06.895151 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:06.895255 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:06.895347 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:06.897583 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:06.898443 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:06.900536 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:06.901436 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:06.901684 (+ 248us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:06.905227 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:06.905237 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:06.905246 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:06.905308 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:06.910334 (+ 5026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:06.911951 (+ 1617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:06.911956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:06.911959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:06.912165 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:06.912872 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:06.912877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:06.912880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:06.913312 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:06.913675 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:06.914208 (+ 533us) server_negotiation.cc:299] Negotiation successful
0923 17:27:06.914812 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":69}
I20260923 17:27:06.919826 18009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:06.931033 18009 catalog_manager.cc:1347] Generated new cluster ID: 92b5ce37af82404795d8c5e135400f6c
I20260923 17:27:06.931218 18009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:06.961862 18009 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:06.963572 18009 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:06.970937 18009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbf330722b574446b1f27be522d840cf: Generated new TSK 0
I20260923 17:27:06.972136 18009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:06.979468 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 17922
I20260923 17:27:06.981161 17984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:27:07.077414 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 17922
W20260923 17:27:07.095680 17907 connection.cc:535] client connection to 127.0.0.1:41213 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41213): Connection reset by peer (error 104)
2026-09-23T17:27:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2148 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:07 dist-test-slave-j0m4 krb5kdc[18021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:07 dist-test-slave-j0m4 krb5kdc[18021](info): set up 2 sockets
Sep 23 17:27:07 dist-test-slave-j0m4 krb5kdc[18021](info): commencing operation
krb5kdc: starting...
W20260923 17:27:08.204304 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:27:08 dist-test-slave-j0m4 krb5kdc[18021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:27:08.480018 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:08.822328 18034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:08.822692 18034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:08.822866 18034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:08.830334 18034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:08.830513 18034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:08.830624 18034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:08.830736 18034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:08.830824 18034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:08.840873 18034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32897
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:08.842290 18034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:08.843710 18034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:08.856014 18042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:08.856107 18039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:08.856065 18040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:08.857456 18034 server_base.cc:1034] running on GCE node
I20260923 17:27:08.858310 18034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:08.859925 18034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:08.861251 18034 hybrid_clock.cc:648] HybridClock initialized: now 1790184428861198 us; error 58 us; skew 500 ppm
I20260923 17:27:08.861676 18034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:08 dist-test-slave-j0m4 krb5kdc[18021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:08.865231 18034 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:27:08.867015 18034 webserver.cc:460] Webserver started at http://127.0.0.1:34621/ using document root <none> and password file <none>
I20260923 17:27:08.867822 18034 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:08.867976 18034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:08.868347 18034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:08.871728 18034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "a033623587294b808685f44e8d5578b1"
format_stamp: "Formatted at 2026-09-23 17:27:08 on dist-test-slave-j0m4"
server_key: "ff6f495d4c32f57339d4c3ee76fe4848"
server_key_iv: "f70889cae5a99b0e4a45c91369109f4a"
server_key_version: "encryptionkey@0"
I20260923 17:27:08.872645 18034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "a033623587294b808685f44e8d5578b1"
format_stamp: "Formatted at 2026-09-23 17:27:08 on dist-test-slave-j0m4"
server_key: "ff6f495d4c32f57339d4c3ee76fe4848"
server_key_iv: "f70889cae5a99b0e4a45c91369109f4a"
server_key_version: "encryptionkey@0"
I20260923 17:27:08.878554 18034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:27:08.882541 18049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:08.884404 18034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:27:08.884647 18034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "a033623587294b808685f44e8d5578b1"
format_stamp: "Formatted at 2026-09-23 17:27:08 on dist-test-slave-j0m4"
server_key: "ff6f495d4c32f57339d4c3ee76fe4848"
server_key_iv: "f70889cae5a99b0e4a45c91369109f4a"
server_key_version: "encryptionkey@0"
I20260923 17:27:08.884927 18034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:08.898756 18034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:08.903533 18034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:08.903837 18034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:08.923326 18034 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32897
I20260923 17:27:08.923293 18100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32897 every 8 connection(s)
I20260923 17:27:08.924786 18034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:08.926622 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 18034
I20260923 17:27:08.926810 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:27:08.927295 28963 external_mini_cluster.cc:1428] Setting key d54563776618df5913fee9c45cd46262
I20260923 17:27:08.929215 18101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:08.941314 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:08.929182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:39503 (local address 127.0.0.1:32897)
0923 17:27:08.930006 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:08.930017 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:08.930057 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:27:08.931734 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:08.931757 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:08.932161 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:08.932509 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:08.933206 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:08.934689 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:08.936612 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:08.937288 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:08.937484 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:08.938818 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:08.938826 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:08.938839 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:08.939490 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:08.939571 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:08.939597 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:08.939722 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:08.940068 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 17:27:08.940273 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":354,"threads_started":1}
I20260923 17:27:08.942440 18101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1: Bootstrap starting.
I20260923 17:27:08.945688 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:08.946369 18101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:08.947484 18101 log.cc:824] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:08.950825 18101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1: No bootstrap required, opened a new log
I20260923 17:27:08.958132 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:08.950290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:57019 (local address 127.0.0.1:32897)
0923 17:27:08.950525 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:08.950531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:08.950866 (+ 335us) server_negotiation.cc:407] Connection header received
0923 17:27:08.951078 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:08.951082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:08.951195 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:08.951276 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:08.951940 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:08.952697 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:08.954714 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:08.955314 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:08.955424 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:08.956825 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:08.956830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:08.956833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:08.956860 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:08.956939 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:08.956964 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:08.957123 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:08.957707 (+ 584us) server_negotiation.cc:299] Negotiation successful
0923 17:27:08.957773 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:27:08.961318 18101 raft_consensus.cc:348] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a033623587294b808685f44e8d5578b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32897 } }
I20260923 17:27:08.961719 18101 raft_consensus.cc:374] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:08.961851 18101 raft_consensus.cc:729] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a033623587294b808685f44e8d5578b1, State: Initialized, Role: FOLLOWER
I20260923 17:27:08.962487 18101 consensus_queue.cc:260] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a033623587294b808685f44e8d5578b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32897 } }
I20260923 17:27:08.962721 18101 raft_consensus.cc:388] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:08.962899 18101 raft_consensus.cc:482] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:08.963092 18101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:08.965418 18101 raft_consensus.cc:504] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a033623587294b808685f44e8d5578b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32897 } }
I20260923 17:27:08.965988 18101 leader_election.cc:302] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a033623587294b808685f44e8d5578b1; no voters:
I20260923 17:27:08.966703 18101 leader_election.cc:288] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:08.966848 18111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:08.967638 18111 raft_consensus.cc:686] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [term 1 LEADER]: Becoming Leader. State: Replica: a033623587294b808685f44e8d5578b1, State: Running, Role: LEADER
I20260923 17:27:08.968293 18111 consensus_queue.cc:237] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a033623587294b808685f44e8d5578b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32897 } }
I20260923 17:27:08.968880 18101 sys_catalog.cc:564] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:08.971365 18113 sys_catalog.cc:455] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a033623587294b808685f44e8d5578b1. Latest consensus state: current_term: 1 leader_uuid: "a033623587294b808685f44e8d5578b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a033623587294b808685f44e8d5578b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32897 } } }
I20260923 17:27:08.971803 18113 sys_catalog.cc:458] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:08.971551 18112 sys_catalog.cc:455] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a033623587294b808685f44e8d5578b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a033623587294b808685f44e8d5578b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32897 } } }
I20260923 17:27:08.971976 18112 sys_catalog.cc:458] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:08.974211 18119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:08.980223 18119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:08.990459 18119 catalog_manager.cc:1347] Generated new cluster ID: 1e842fed77a34972a7b4483cc489c9db
I20260923 17:27:08.990599 18119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:09.022415 18119 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:09.024413 18119 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:09.041416 18119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a033623587294b808685f44e8d5578b1: Generated new TSK 0
I20260923 17:27:09.042569 18119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:09.061447 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18034
W20260923 17:27:09.062872 18096 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.197:39503
W20260923 17:27:09.063814 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18034: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:27:09.063972 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 18034
W20260923 17:27:09.081550 18019 connection.cc:535] client connection to 127.0.0.1:32897 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32897): Connection reset by peer (error 104)
2026-09-23T17:27:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1981 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:09 dist-test-slave-j0m4 krb5kdc[18132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:09 dist-test-slave-j0m4 krb5kdc[18132](info): set up 2 sockets
Sep 23 17:27:09 dist-test-slave-j0m4 krb5kdc[18132](info): commencing operation
krb5kdc: starting...
W20260923 17:27:10.175711 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:27:10 dist-test-slave-j0m4 krb5kdc[18132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:27:10.451565 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:10.797609 18145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:10.797969 18145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:10.798130 18145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:10.806039 18145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:10.806186 18145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:10.806236 18145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:10.806383 18145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:10.806428 18145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:10.817411 18145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45497
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:10.818837 18145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:10.820219 18145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:10.832973 18153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:10.832984 18151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:10.833851 18150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:10.834010 18145 server_base.cc:1034] running on GCE node
I20260923 17:27:10.834910 18145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:10.836465 18145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:10.837702 18145 hybrid_clock.cc:648] HybridClock initialized: now 1790184430837657 us; error 39 us; skew 500 ppm
I20260923 17:27:10.838083 18145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:10 dist-test-slave-j0m4 krb5kdc[18132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:10.848172 18145 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:27:10.849974 18145 webserver.cc:460] Webserver started at http://127.0.0.1:38955/ using document root <none> and password file <none>
I20260923 17:27:10.850742 18145 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:10.850914 18145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:10.851312 18145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:10.854563 18145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "193761db96184f1b8f892ed5d6640acf"
format_stamp: "Formatted at 2026-09-23 17:27:10 on dist-test-slave-j0m4"
server_key: "be97f8f442ef7bdf872fd024fe0bca8b"
server_key_iv: "6a216b5fff4864a0040c4b6245e4c412"
server_key_version: "encryptionkey@0"
I20260923 17:27:10.855492 18145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "193761db96184f1b8f892ed5d6640acf"
format_stamp: "Formatted at 2026-09-23 17:27:10 on dist-test-slave-j0m4"
server_key: "be97f8f442ef7bdf872fd024fe0bca8b"
server_key_iv: "6a216b5fff4864a0040c4b6245e4c412"
server_key_version: "encryptionkey@0"
I20260923 17:27:10.861267 18145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:27:10.865137 18160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:10.866663 18145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:27:10.866891 18145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "193761db96184f1b8f892ed5d6640acf"
format_stamp: "Formatted at 2026-09-23 17:27:10 on dist-test-slave-j0m4"
server_key: "be97f8f442ef7bdf872fd024fe0bca8b"
server_key_iv: "6a216b5fff4864a0040c4b6245e4c412"
server_key_version: "encryptionkey@0"
I20260923 17:27:10.867117 18145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:10.873567 18145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:10.878175 18145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:10.878523 18145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:10.898752 18145 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45497
I20260923 17:27:10.898763 18211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45497 every 8 connection(s)
I20260923 17:27:10.900367 18145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:10.904847 18212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:10.909042 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 18145
I20260923 17:27:10.909241 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:27:10.909807 28963 external_mini_cluster.cc:1428] Setting key 94bdd2de68c551f5ad05fa0ed421e0a1
I20260923 17:27:10.917932 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:10.911884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:56489 (local address 127.0.0.1:45497)
0923 17:27:10.912686 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:10.912698 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:10.912740 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:27:10.914440 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:10.914463 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:10.914837 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:10.915269 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:10.916313 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:10.916319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:10.916336 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:10.916357 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:10.916415 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:10.916436 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:10.916609 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:10.916718 (+ 109us) server_negotiation.cc:299] Negotiation successful
0923 17:27:10.916977 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":372,"threads_started":1}
I20260923 17:27:10.918663 18212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf: Bootstrap starting.
I20260923 17:27:10.921806 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:10.922166 18212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:10.923477 18212 log.cc:824] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:10.926965 18212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf: No bootstrap required, opened a new log
I20260923 17:27:10.929940 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:10.927226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:59095 (local address 127.0.0.1:45497)
0923 17:27:10.927439 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:10.927445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:10.927803 (+ 358us) server_negotiation.cc:407] Connection header received
0923 17:27:10.928069 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:10.928073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:10.928121 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:10.928204 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:10.929180 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:10.929184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:10.929188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:10.929205 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:10.929244 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:10.929259 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:10.929361 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:10.929519 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 17:27:10.929574 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:27:10.936373 18212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193761db96184f1b8f892ed5d6640acf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260923 17:27:10.936841 18212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:10.936956 18212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 193761db96184f1b8f892ed5d6640acf, State: Initialized, Role: FOLLOWER
I20260923 17:27:10.937553 18212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193761db96184f1b8f892ed5d6640acf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260923 17:27:10.937803 18212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:10.937959 18212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:10.938140 18212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:10.940626 18212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193761db96184f1b8f892ed5d6640acf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260923 17:27:10.941246 18212 leader_election.cc:302] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 193761db96184f1b8f892ed5d6640acf; no voters:
I20260923 17:27:10.942106 18212 leader_election.cc:288] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:10.942219 18222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:10.942886 18222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [term 1 LEADER]: Becoming Leader. State: Replica: 193761db96184f1b8f892ed5d6640acf, State: Running, Role: LEADER
I20260923 17:27:10.943586 18222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193761db96184f1b8f892ed5d6640acf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260923 17:27:10.944684 18212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:10.946841 18224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "193761db96184f1b8f892ed5d6640acf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193761db96184f1b8f892ed5d6640acf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } } }
I20260923 17:27:10.947307 18224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:10.948062 18223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 193761db96184f1b8f892ed5d6640acf. Latest consensus state: current_term: 1 leader_uuid: "193761db96184f1b8f892ed5d6640acf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193761db96184f1b8f892ed5d6640acf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } } }
I20260923 17:27:10.948438 18223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:10.948962 18230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:10.954463 18230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:10.964376 18230 catalog_manager.cc:1347] Generated new cluster ID: 9b8d270e764745ecb6748c7d71cbb7d9
I20260923 17:27:10.964609 18230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:11.002928 18230 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:11.003477 18230 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:11.019450 18230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 193761db96184f1b8f892ed5d6640acf: Generated new TSK 0
I20260923 17:27:11.020715 18230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:11.038597 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18145
W20260923 17:27:11.039839 18207 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.197:56489
W20260923 17:27:11.040663 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18145: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:27:11.040829 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 18145
2026-09-23T17:27:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:11 dist-test-slave-j0m4 krb5kdc[18243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:11 dist-test-slave-j0m4 krb5kdc[18243](info): set up 2 sockets
Sep 23 17:27:11 dist-test-slave-j0m4 krb5kdc[18243](info): commencing operation
krb5kdc: starting...
W20260923 17:27:12.187793 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:27:12 dist-test-slave-j0m4 krb5kdc[18243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:27:12.466969 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:41841
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:41841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:12.813627 18256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:12.814018 18256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:12.814182 18256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:12.821806 18256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:12.821992 18256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:12.822098 18256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:12.822217 18256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:12.822302 18256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:12.833082 18256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:41841
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:41841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:12.834561 18256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:12.835966 18256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:12.849141 18264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:12.849223 18262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:12.849185 18261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:12.850001 18256 server_base.cc:1034] running on GCE node
I20260923 17:27:12.851039 18256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:12.852715 18256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:12.853991 18256 hybrid_clock.cc:648] HybridClock initialized: now 1790184432853936 us; error 47 us; skew 500 ppm
I20260923 17:27:12.854502 18256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:12 dist-test-slave-j0m4 krb5kdc[18243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184432, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:12.858147 18256 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:27:12.859970 18256 webserver.cc:460] Webserver started at http://127.28.72.254:34825/ using document root <none> and password file <none>
I20260923 17:27:12.860706 18256 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:12.860859 18256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:12.861196 18256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:12.864527 18256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "53d56f37b99b4554905889e9e03d87c0"
format_stamp: "Formatted at 2026-09-23 17:27:12 on dist-test-slave-j0m4"
server_key: "8e5a66d4c2937aa897d576cf61e73b78"
server_key_iv: "07eba904f66c94095d7c2611c5d483e0"
server_key_version: "encryptionkey@0"
I20260923 17:27:12.865366 18256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "53d56f37b99b4554905889e9e03d87c0"
format_stamp: "Formatted at 2026-09-23 17:27:12 on dist-test-slave-j0m4"
server_key: "8e5a66d4c2937aa897d576cf61e73b78"
server_key_iv: "07eba904f66c94095d7c2611c5d483e0"
server_key_version: "encryptionkey@0"
I20260923 17:27:12.871188 18256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:27:12.875087 18271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:12.876778 18256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:27:12.877012 18256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "53d56f37b99b4554905889e9e03d87c0"
format_stamp: "Formatted at 2026-09-23 17:27:12 on dist-test-slave-j0m4"
server_key: "8e5a66d4c2937aa897d576cf61e73b78"
server_key_iv: "07eba904f66c94095d7c2611c5d483e0"
server_key_version: "encryptionkey@0"
I20260923 17:27:12.877250 18256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:12.903172 18256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:12.907991 18256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:12.908314 18256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:12.928707 18256 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:41841
I20260923 17:27:12.928748 18322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:41841 every 8 connection(s)
I20260923 17:27:12.930158 18256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:12.934360 18323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:12.935007 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 18256
I20260923 17:27:12.935232 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:27:12.935746 28963 external_mini_cluster.cc:1428] Setting key a4704cfee8b95082bdff5ce54bcd1152
I20260923 17:27:12.950160 18323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0: Bootstrap starting.
Sep 23 17:27:12 dist-test-slave-j0m4 krb5kdc[18243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:27:12.954072 18323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:12.955539 18323 log.cc:824] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:12.958971 18323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0: No bootstrap required, opened a new log
I20260923 17:27:12.962165 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.937707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46937 (local address 127.28.72.254:41841)
0923 17:27:12.938542 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:12.938557 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:12.938597 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:27:12.940815 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:12.940847 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:12.941286 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:12.941663 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:12.943845 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.945270 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.947461 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.948174 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.948492 (+ 318us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.953605 (+ 5113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:12.953614 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:12.953637 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:12.953679 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:12.957355 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:12.958347 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:12.958354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:12.958361 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:12.958559 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:12.959123 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:12.959149 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:12.959152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:12.959752 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:12.960137 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:12.960671 (+ 534us) server_negotiation.cc:299] Negotiation successful
0923 17:27:12.961040 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":604,"thread_start_us":386,"threads_started":1}
I20260923 17:27:12.966746 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:12.969082 18323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d56f37b99b4554905889e9e03d87c0" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 41841 } }
I20260923 17:27:12.969540 18323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:12.969666 18323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d56f37b99b4554905889e9e03d87c0, State: Initialized, Role: FOLLOWER
I20260923 17:27:12.970425 18323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d56f37b99b4554905889e9e03d87c0" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 41841 } }
I20260923 17:27:12.970705 18323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:12.970872 18323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:12.971058 18323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:12.973945 18323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d56f37b99b4554905889e9e03d87c0" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 41841 } }
I20260923 17:27:12.974584 18323 leader_election.cc:302] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53d56f37b99b4554905889e9e03d87c0; no voters:
I20260923 17:27:12.975368 18323 leader_election.cc:288] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:12.975538 18333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:12.976260 18333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [term 1 LEADER]: Becoming Leader. State: Replica: 53d56f37b99b4554905889e9e03d87c0, State: Running, Role: LEADER
I20260923 17:27:12.976925 18333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d56f37b99b4554905889e9e03d87c0" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 41841 } }
I20260923 17:27:12.979082 18333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53d56f37b99b4554905889e9e03d87c0. Latest consensus state: current_term: 1 leader_uuid: "53d56f37b99b4554905889e9e03d87c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d56f37b99b4554905889e9e03d87c0" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 41841 } } }
I20260923 17:27:12.980638 18333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:12.980122 18335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53d56f37b99b4554905889e9e03d87c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d56f37b99b4554905889e9e03d87c0" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 41841 } } }
I20260923 17:27:12.982527 18335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:12.984318 18323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:12.991760 18337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:12.992481 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.971538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51557 (local address 127.28.72.254:41841)
0923 17:27:12.971756 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:12.971764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:12.972281 (+ 517us) server_negotiation.cc:407] Connection header received
0923 17:27:12.972441 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:12.972447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:12.972545 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:12.972664 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:12.974246 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.975176 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.977519 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.978220 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.978349 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.982922 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:12.982931 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:12.982935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:12.982980 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:12.987660 (+ 4680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:12.989136 (+ 1476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:12.989146 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:12.989153 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:12.989412 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:12.990175 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:12.990183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:12.990189 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:12.990734 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:12.991163 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:12.991667 (+ 504us) server_negotiation.cc:299] Negotiation successful
0923 17:27:12.991952 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:27:12.997934 18337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:13.007917 18337 catalog_manager.cc:1347] Generated new cluster ID: 6a6e6eb4c6ef4c27b8ec95ad59ab31e6
I20260923 17:27:13.008068 18337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:13.030601 18337 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:13.032116 18337 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:13.042377 18337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53d56f37b99b4554905889e9e03d87c0: Generated new TSK 0
I20260923 17:27:13.043394 18337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:13.060125 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18256
I20260923 17:27:13.061484 18318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:27:13.156965 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 18256
W20260923 17:27:13.174645 18241 connection.cc:535] client connection to 127.28.72.254:41841 recv error: Network error: failed to read from TLS socket (remote: 127.28.72.254:41841): Connection reset by peer (error 104)
2026-09-23T17:27:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2115 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:13 dist-test-slave-j0m4 krb5kdc[18355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:13 dist-test-slave-j0m4 krb5kdc[18355](info): set up 2 sockets
Sep 23 17:27:13 dist-test-slave-j0m4 krb5kdc[18355](info): commencing operation
krb5kdc: starting...
W20260923 17:27:14.276072 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:27:14 dist-test-slave-j0m4 krb5kdc[18355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:27:14.555508 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:33913
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:33913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:14.900424 18368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:14.900720 18368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:14.900914 18368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:14.908504 18368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:14.908630 18368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:14.908736 18368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:14.908830 18368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:14.908882 18368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:14.918982 18368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:33913
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:33913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:14.920450 18368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:14.921845 18368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:14.934144 18373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:14.934376 18374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:14.934424 18376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:14.935084 18368 server_base.cc:1034] running on GCE node
I20260923 17:27:14.936014 18368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:14.937516 18368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:14.938756 18368 hybrid_clock.cc:648] HybridClock initialized: now 1790184434938689 us; error 60 us; skew 500 ppm
I20260923 17:27:14.939189 18368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:14 dist-test-slave-j0m4 krb5kdc[18355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184434, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:14.942744 18368 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:27:14.944368 18368 webserver.cc:460] Webserver started at http://127.28.72.254:43231/ using document root <none> and password file <none>
I20260923 17:27:14.945084 18368 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:14.945190 18368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:14.945451 18368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:14.948611 18368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "712b638518f5464785ba093a4edec2f3"
format_stamp: "Formatted at 2026-09-23 17:27:14 on dist-test-slave-j0m4"
server_key: "ecb73e10e518ca942bced6d5a7f2adbc"
server_key_iv: "e6a7040aa7bf61573083154867adcbed"
server_key_version: "encryptionkey@0"
I20260923 17:27:14.949383 18368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "712b638518f5464785ba093a4edec2f3"
format_stamp: "Formatted at 2026-09-23 17:27:14 on dist-test-slave-j0m4"
server_key: "ecb73e10e518ca942bced6d5a7f2adbc"
server_key_iv: "e6a7040aa7bf61573083154867adcbed"
server_key_version: "encryptionkey@0"
I20260923 17:27:14.955219 18368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:27:14.959148 18383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:14.960587 18368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 17:27:14.960783 18368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "712b638518f5464785ba093a4edec2f3"
format_stamp: "Formatted at 2026-09-23 17:27:14 on dist-test-slave-j0m4"
server_key: "ecb73e10e518ca942bced6d5a7f2adbc"
server_key_iv: "e6a7040aa7bf61573083154867adcbed"
server_key_version: "encryptionkey@0"
I20260923 17:27:14.961022 18368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:14.978363 18368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:14.982932 18368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:14.983212 18368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:15.002243 18368 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:33913
I20260923 17:27:15.002262 18434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:33913 every 8 connection(s)
I20260923 17:27:15.003775 18368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:15.007709 18435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:15.012586 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 18368
I20260923 17:27:15.012765 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:27:15.013199 28963 external_mini_cluster.cc:1428] Setting key c69d143acf32e0be01e4fcff8dd88796
I20260923 17:27:15.020831 18435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3: Bootstrap starting.
I20260923 17:27:15.024848 18435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:15.026177 18435 log.cc:824] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:15.029647 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.015204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40613 (local address 127.28.72.254:33913)
0923 17:27:15.016102 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.016119 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.016170 (+ 51us) server_negotiation.cc:407] Connection header received
0923 17:27:15.018902 (+ 2732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.018938 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.019566 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.020056 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.020628 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.022654 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.024143 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.025251 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.025489 (+ 238us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.026755 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.026773 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.026800 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:15.027521 (+ 721us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.027640 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:15.027672 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.027837 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.028169 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.028371 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":378,"threads_started":1}
I20260923 17:27:15.029731 18435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3: No bootstrap required, opened a new log
I20260923 17:27:15.034099 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:15.038168 18435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "712b638518f5464785ba093a4edec2f3" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 33913 } }
I20260923 17:27:15.038600 18435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:15.038722 18435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 712b638518f5464785ba093a4edec2f3, State: Initialized, Role: FOLLOWER
I20260923 17:27:15.039456 18435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "712b638518f5464785ba093a4edec2f3" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 33913 } }
I20260923 17:27:15.039710 18435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:15.039871 18435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:15.040073 18435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:15.043030 18435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "712b638518f5464785ba093a4edec2f3" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 33913 } }
I20260923 17:27:15.043624 18435 leader_election.cc:302] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 712b638518f5464785ba093a4edec2f3; no voters:
I20260923 17:27:15.044332 18435 leader_election.cc:288] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:15.044472 18445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:15.045342 18445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [term 1 LEADER]: Becoming Leader. State: Replica: 712b638518f5464785ba093a4edec2f3, State: Running, Role: LEADER
I20260923 17:27:15.046151 18445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "712b638518f5464785ba093a4edec2f3" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 33913 } }
I20260923 17:27:15.047528 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.039054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35919 (local address 127.28.72.254:33913)
0923 17:27:15.039311 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.039319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.039955 (+ 636us) server_negotiation.cc:407] Connection header received
0923 17:27:15.040178 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.040183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.040270 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.040401 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.040898 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.042059 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.043737 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.044503 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.044652 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.046282 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.046291 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.046300 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:15.046340 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.046435 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:15.046459 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.046591 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.047090 (+ 499us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.047177 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260923 17:27:15.048203 18435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:15.050513 18446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 712b638518f5464785ba093a4edec2f3. Latest consensus state: current_term: 1 leader_uuid: "712b638518f5464785ba093a4edec2f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "712b638518f5464785ba093a4edec2f3" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 33913 } } }
I20260923 17:27:15.050541 18447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "712b638518f5464785ba093a4edec2f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "712b638518f5464785ba093a4edec2f3" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 33913 } } }
I20260923 17:27:15.051054 18446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:15.051216 18447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:15.052809 18450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:15.059002 18450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:15.070866 18450 catalog_manager.cc:1347] Generated new cluster ID: 2e8c9e7881a24e2baf975be8efaaddb7
I20260923 17:27:15.070979 18450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:15.095952 18450 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:15.097419 18450 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:15.110617 18450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 712b638518f5464785ba093a4edec2f3: Generated new TSK 0
I20260923 17:27:15.111820 18450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:15.128410 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18368
W20260923 17:27:15.129849 18430 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40613
W20260923 17:27:15.130779 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18368: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:27:15.130975 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 18368
W20260923 17:27:15.149377 18353 connection.cc:535] client connection to 127.28.72.254:33913 recv error: Network error: failed to read from TLS socket (remote: 127.28.72.254:33913): Connection reset by peer (error 104)
2026-09-23T17:27:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:15 dist-test-slave-j0m4 krb5kdc[18466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:15 dist-test-slave-j0m4 krb5kdc[18466](info): set up 2 sockets
Sep 23 17:27:15 dist-test-slave-j0m4 krb5kdc[18466](info): commencing operation
krb5kdc: starting...
W20260923 17:27:16.295840 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:27:16 dist-test-slave-j0m4 krb5kdc[18466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:27:16.579762 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:43141
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:35981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:43141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:16.935341 18479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:16.935676 18479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:16.935844 18479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:16.943321 18479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:16.943451 18479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:16.943573 18479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:16.943686 18479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:16.943756 18479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:16.954079 18479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:35981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:43141
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:43141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:16.955547 18479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:16.957007 18479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:16.969676 18487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:16.969722 18485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:16.969642 18484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:16.970590 18479 server_base.cc:1034] running on GCE node
I20260923 17:27:16.971611 18479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:16.973348 18479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:16.974620 18479 hybrid_clock.cc:648] HybridClock initialized: now 1790184436974565 us; error 44 us; skew 500 ppm
I20260923 17:27:16.975189 18479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:16 dist-test-slave-j0m4 krb5kdc[18466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184436, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:16.979027 18479 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:27:16.981086 18479 webserver.cc:460] Webserver started at http://127.28.72.254:33117/ using document root <none> and password file <none>
I20260923 17:27:16.981922 18479 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:16.982096 18479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:16.982461 18479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:16.986048 18479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "dc6ebbc667464182a90bc5060f27c213"
format_stamp: "Formatted at 2026-09-23 17:27:16 on dist-test-slave-j0m4"
server_key: "847ab2c17283d39c93f05a1fbd1327e5"
server_key_iv: "019a883acf13677e8fec8e47afba866f"
server_key_version: "encryptionkey@0"
I20260923 17:27:16.987005 18479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "dc6ebbc667464182a90bc5060f27c213"
format_stamp: "Formatted at 2026-09-23 17:27:16 on dist-test-slave-j0m4"
server_key: "847ab2c17283d39c93f05a1fbd1327e5"
server_key_iv: "019a883acf13677e8fec8e47afba866f"
server_key_version: "encryptionkey@0"
I20260923 17:27:16.993139 18479 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260923 17:27:16.997406 18494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:16.999003 18479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:27:16.999236 18479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "dc6ebbc667464182a90bc5060f27c213"
format_stamp: "Formatted at 2026-09-23 17:27:16 on dist-test-slave-j0m4"
server_key: "847ab2c17283d39c93f05a1fbd1327e5"
server_key_iv: "019a883acf13677e8fec8e47afba866f"
server_key_version: "encryptionkey@0"
I20260923 17:27:16.999472 18479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:17.006201 18479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:17.010733 18479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:17.011008 18479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:17.030486 18479 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:43141
I20260923 17:27:17.030535 18545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:43141 every 8 connection(s)
I20260923 17:27:17.032007 18479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:17.035945 18546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:17.038101 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 18479
I20260923 17:27:17.038302 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:27:17.038782 28963 external_mini_cluster.cc:1428] Setting key ae5098eb58a9f9b6b9da703597390dcf
I20260923 17:27:17.047400 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:17.040783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52079 (local address 127.28.72.254:43141)
0923 17:27:17.041719 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:17.041741 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:17.041786 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:27:17.043470 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:17.043495 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:17.044030 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:17.044465 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:17.045561 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:17.045568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:17.045581 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:17.045598 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:17.045668 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:17.045690 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:17.045872 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:17.046021 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 17:27:17.046247 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":511,"threads_started":1}
I20260923 17:27:17.050928 18546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213: Bootstrap starting.
I20260923 17:27:17.051607 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:17.054260 18546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:17.055603 18546 log.cc:824] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:17.058722 18546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213: No bootstrap required, opened a new log
I20260923 17:27:17.059957 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:17.056604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40243 (local address 127.28.72.254:43141)
0923 17:27:17.056877 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:17.056903 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:17.057534 (+ 631us) server_negotiation.cc:407] Connection header received
0923 17:27:17.057778 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:17.057785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:17.057860 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:17.057981 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:17.059161 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:17.059172 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:17.059177 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:17.059200 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:17.059260 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:17.059278 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:17.059433 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:17.059535 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 17:27:17.059606 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:27:17.066646 18546 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6ebbc667464182a90bc5060f27c213" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43141 } }
I20260923 17:27:17.067144 18546 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:17.067276 18546 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6ebbc667464182a90bc5060f27c213, State: Initialized, Role: FOLLOWER
I20260923 17:27:17.067929 18546 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6ebbc667464182a90bc5060f27c213" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43141 } }
I20260923 17:27:17.068194 18546 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:17.068372 18546 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:17.068545 18546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:17.070895 18546 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6ebbc667464182a90bc5060f27c213" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43141 } }
I20260923 17:27:17.071491 18546 leader_election.cc:302] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc6ebbc667464182a90bc5060f27c213; no voters:
I20260923 17:27:17.072111 18546 leader_election.cc:288] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:17.072259 18556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:17.073051 18556 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [term 1 LEADER]: Becoming Leader. State: Replica: dc6ebbc667464182a90bc5060f27c213, State: Running, Role: LEADER
I20260923 17:27:17.073644 18556 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6ebbc667464182a90bc5060f27c213" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43141 } }
I20260923 17:27:17.074301 18546 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:17.076735 18557 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc6ebbc667464182a90bc5060f27c213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6ebbc667464182a90bc5060f27c213" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43141 } } }
I20260923 17:27:17.077250 18557 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:17.076778 18558 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc6ebbc667464182a90bc5060f27c213. Latest consensus state: current_term: 1 leader_uuid: "dc6ebbc667464182a90bc5060f27c213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6ebbc667464182a90bc5060f27c213" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 43141 } } }
I20260923 17:27:17.077941 18558 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:17.080269 18564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:17.085012 18564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:17.094592 18564 catalog_manager.cc:1347] Generated new cluster ID: 2080243cdde94dada2dac902c1bbb667
I20260923 17:27:17.094772 18564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:17.115392 18564 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:17.115885 18564 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:17.126778 18564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc6ebbc667464182a90bc5060f27c213: Generated new TSK 0
I20260923 17:27:17.127898 18564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:17.144675 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18479
W20260923 17:27:17.145967 18541 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52079
W20260923 17:27:17.146816 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18479: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:27:17.147042 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 18479
2026-09-23T17:27:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2015 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:17 dist-test-slave-j0m4 krb5kdc[18577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:17 dist-test-slave-j0m4 krb5kdc[18577](info): set up 2 sockets
Sep 23 17:27:17 dist-test-slave-j0m4 krb5kdc[18577](info): commencing operation
krb5kdc: starting...
W20260923 17:27:18.271798 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:27:18 dist-test-slave-j0m4 krb5kdc[18577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:27:18.551470 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:32847
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:44413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:32847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:18.902072 18590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:18.902395 18590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:18.902501 18590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:18.910259 18590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:18.910382 18590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:18.910436 18590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:18.910498 18590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:18.910544 18590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:18.920795 18590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:44413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:32847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:32847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:18.922187 18590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:18.923494 18590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:18.936198 18595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:18.936234 18598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:18.936198 18596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:18.937342 18590 server_base.cc:1034] running on GCE node
I20260923 17:27:18.938230 18590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:18.939788 18590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:18.941025 18590 hybrid_clock.cc:648] HybridClock initialized: now 1790184438940956 us; error 54 us; skew 500 ppm
I20260923 17:27:18.941442 18590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:18 dist-test-slave-j0m4 krb5kdc[18577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184438, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:18.944787 18590 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:27:18.946494 18590 webserver.cc:460] Webserver started at http://127.28.72.254:38017/ using document root <none> and password file <none>
I20260923 17:27:18.947305 18590 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:18.947434 18590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:18.947765 18590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:18.950932 18590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "a131cc6e9fb04b3cb48c2c433ad38c29"
format_stamp: "Formatted at 2026-09-23 17:27:18 on dist-test-slave-j0m4"
server_key: "9be742d10cd89bf26642968e2b3c623a"
server_key_iv: "db91857fd9258ad5e550f7785a68cd9d"
server_key_version: "encryptionkey@0"
I20260923 17:27:18.951797 18590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "a131cc6e9fb04b3cb48c2c433ad38c29"
format_stamp: "Formatted at 2026-09-23 17:27:18 on dist-test-slave-j0m4"
server_key: "9be742d10cd89bf26642968e2b3c623a"
server_key_iv: "db91857fd9258ad5e550f7785a68cd9d"
server_key_version: "encryptionkey@0"
I20260923 17:27:18.957456 18590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 17:27:18.961848 18605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:18.963614 18590 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260923 17:27:18.963788 18590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "a131cc6e9fb04b3cb48c2c433ad38c29"
format_stamp: "Formatted at 2026-09-23 17:27:18 on dist-test-slave-j0m4"
server_key: "9be742d10cd89bf26642968e2b3c623a"
server_key_iv: "db91857fd9258ad5e550f7785a68cd9d"
server_key_version: "encryptionkey@0"
I20260923 17:27:18.963992 18590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:18.979533 18590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:18.984736 18590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:18.985005 18590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:19.005070 18590 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:32847
I20260923 17:27:19.005092 18656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:32847 every 8 connection(s)
I20260923 17:27:19.006644 18590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:19.009202 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 18590
I20260923 17:27:19.009395 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:27:19.009974 28963 external_mini_cluster.cc:1428] Setting key b1cd68fb26f2b1d84c68bca401164810
I20260923 17:27:19.011730 18657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:27:19 dist-test-slave-j0m4 krb5kdc[18577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:27:19.026731 18657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29: Bootstrap starting.
I20260923 17:27:19.030213 18657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:19.031383 18657 log.cc:824] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:19.034880 18657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29: No bootstrap required, opened a new log
I20260923 17:27:19.035362 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:19.012086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49707 (local address 127.28.72.254:32847)
0923 17:27:19.012961 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:19.012973 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:19.013017 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:27:19.014692 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:19.014718 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:19.015230 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:19.015599 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:19.017448 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:19.018969 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:19.020843 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:19.021603 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:19.021754 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:19.026982 (+ 5228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:19.026992 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:19.027010 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:19.027061 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:19.030491 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:19.031364 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:19.031375 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:19.031383 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:19.031616 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:19.032130 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:19.032137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:19.032139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:19.032760 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:19.033064 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:19.033477 (+ 413us) server_negotiation.cc:299] Negotiation successful
0923 17:27:19.033944 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":380,"threads_started":1}
I20260923 17:27:19.039795 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:19.045595 18657 raft_consensus.cc:348] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131cc6e9fb04b3cb48c2c433ad38c29" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 32847 } }
I20260923 17:27:19.046116 18657 raft_consensus.cc:374] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:19.046315 18657 raft_consensus.cc:729] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a131cc6e9fb04b3cb48c2c433ad38c29, State: Initialized, Role: FOLLOWER
I20260923 17:27:19.047035 18657 consensus_queue.cc:260] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131cc6e9fb04b3cb48c2c433ad38c29" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 32847 } }
I20260923 17:27:19.047407 18657 raft_consensus.cc:388] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:19.047582 18657 raft_consensus.cc:482] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:19.047788 18657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:19.050154 18657 raft_consensus.cc:504] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131cc6e9fb04b3cb48c2c433ad38c29" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 32847 } }
I20260923 17:27:19.050738 18657 leader_election.cc:302] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a131cc6e9fb04b3cb48c2c433ad38c29; no voters:
I20260923 17:27:19.051476 18657 leader_election.cc:288] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:19.051607 18667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:19.053138 18667 raft_consensus.cc:686] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [term 1 LEADER]: Becoming Leader. State: Replica: a131cc6e9fb04b3cb48c2c433ad38c29, State: Running, Role: LEADER
I20260923 17:27:19.053849 18667 consensus_queue.cc:237] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131cc6e9fb04b3cb48c2c433ad38c29" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 32847 } }
I20260923 17:27:19.054109 18657 sys_catalog.cc:564] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:19.061658 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:19.045919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51407 (local address 127.28.72.254:32847)
0923 17:27:19.046150 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:19.046157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:19.046743 (+ 586us) server_negotiation.cc:407] Connection header received
0923 17:27:19.047017 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:19.047021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:19.047095 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:19.047276 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:19.048950 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:19.049746 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:19.051233 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:19.051890 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:19.052720 (+ 830us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:19.054898 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:19.054903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:19.054907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:19.054940 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:19.058191 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:19.059246 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:19.059250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:19.059252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:19.059380 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:19.059761 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:19.059768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:19.059773 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:19.060120 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:19.060422 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:19.060853 (+ 431us) server_negotiation.cc:299] Negotiation successful
0923 17:27:19.061073 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:27:19.063907 18668 sys_catalog.cc:455] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a131cc6e9fb04b3cb48c2c433ad38c29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131cc6e9fb04b3cb48c2c433ad38c29" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 32847 } } }
I20260923 17:27:19.064286 18668 sys_catalog.cc:458] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:19.065168 18674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:19.065025 18669 sys_catalog.cc:455] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a131cc6e9fb04b3cb48c2c433ad38c29. Latest consensus state: current_term: 1 leader_uuid: "a131cc6e9fb04b3cb48c2c433ad38c29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131cc6e9fb04b3cb48c2c433ad38c29" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 32847 } } }
I20260923 17:27:19.065414 18669 sys_catalog.cc:458] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:19.074275 18674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:19.085625 18674 catalog_manager.cc:1347] Generated new cluster ID: 9f42b07c9ac94567a6573948bbed3007
I20260923 17:27:19.085780 18674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:19.096858 18674 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:19.098393 18674 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:19.110862 18674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a131cc6e9fb04b3cb48c2c433ad38c29: Generated new TSK 0
I20260923 17:27:19.112046 18674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:19.118696 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18590
I20260923 17:27:19.120015 18652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:27:19.213562 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 18590
2026-09-23T17:27:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2066 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:19 dist-test-slave-j0m4 krb5kdc[18689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:19 dist-test-slave-j0m4 krb5kdc[18689](info): set up 2 sockets
Sep 23 17:27:19 dist-test-slave-j0m4 krb5kdc[18689](info): commencing operation
krb5kdc: starting...
W20260923 17:27:20.347815 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.006s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:27:20 dist-test-slave-j0m4 krb5kdc[18689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:27:20.634848 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:40177
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:36071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:40177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:20.987025 18702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:20.987399 18702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:20.987579 18702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:20.995290 18702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:20.995424 18702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:20.995471 18702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:20.995597 18702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:20.995692 18702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:21.006201 18702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:36071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:40177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:40177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:21.007768 18702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:21.009148 18702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:21.022342 18707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:21.022375 18708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:21.022604 18710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:21.023425 18702 server_base.cc:1034] running on GCE node
I20260923 17:27:21.024288 18702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:21.025900 18702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:21.027208 18702 hybrid_clock.cc:648] HybridClock initialized: now 1790184441027165 us; error 53 us; skew 500 ppm
I20260923 17:27:21.027595 18702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:21 dist-test-slave-j0m4 krb5kdc[18689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184441, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:21.036283 18702 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:27:21.038036 18702 webserver.cc:460] Webserver started at http://127.28.72.254:34013/ using document root <none> and password file <none>
I20260923 17:27:21.038806 18702 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:21.038968 18702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:21.039356 18702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:21.042642 18702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "5bbca88470964e818e0c57e98ca5bc95"
format_stamp: "Formatted at 2026-09-23 17:27:21 on dist-test-slave-j0m4"
server_key: "e7e68f7aa22264342f89bbaeddca0a03"
server_key_iv: "5fdaaf79241bc40b450e511fd5667c6d"
server_key_version: "encryptionkey@0"
I20260923 17:27:21.043534 18702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "5bbca88470964e818e0c57e98ca5bc95"
format_stamp: "Formatted at 2026-09-23 17:27:21 on dist-test-slave-j0m4"
server_key: "e7e68f7aa22264342f89bbaeddca0a03"
server_key_iv: "5fdaaf79241bc40b450e511fd5667c6d"
server_key_version: "encryptionkey@0"
I20260923 17:27:21.049352 18702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:27:21.053295 18717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:21.054831 18702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:27:21.055040 18702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "5bbca88470964e818e0c57e98ca5bc95"
format_stamp: "Formatted at 2026-09-23 17:27:21 on dist-test-slave-j0m4"
server_key: "e7e68f7aa22264342f89bbaeddca0a03"
server_key_iv: "5fdaaf79241bc40b450e511fd5667c6d"
server_key_version: "encryptionkey@0"
I20260923 17:27:21.055344 18702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:21.072027 18702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:21.077065 18702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:21.077412 18702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:21.097889 18702 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:40177
I20260923 17:27:21.097929 18768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:40177 every 8 connection(s)
I20260923 17:27:21.099413 18702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:21.103423 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 18702
I20260923 17:27:21.103463 18769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:21.103622 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:27:21.104080 28963 external_mini_cluster.cc:1428] Setting key cdcca55088084e1e05a39184f7e02029
I20260923 17:27:21.118927 18769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95: Bootstrap starting.
I20260923 17:27:21.120007 18772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.106107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55571 (local address 127.28.72.254:40177)
0923 17:27:21.107207 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:21.107224 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:21.107274 (+ 50us) server_negotiation.cc:407] Connection header received
0923 17:27:21.108879 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:21.108906 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:21.109383 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:21.109763 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:21.110497 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.112036 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.114014 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.115060 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.115303 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.116883 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:21.116892 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:21.116910 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:21.117721 (+ 811us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:21.117822 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:21.117868 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:21.117995 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:21.118355 (+ 360us) server_negotiation.cc:299] Negotiation successful
0923 17:27:21.118711 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":505,"threads_started":1}
I20260923 17:27:21.122792 18769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:21.124078 18769 log.cc:824] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:21.124482 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:21.127764 18769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95: No bootstrap required, opened a new log
I20260923 17:27:21.137756 18772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.130119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55593 (local address 127.28.72.254:40177)
0923 17:27:21.130374 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:21.130383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:21.130982 (+ 599us) server_negotiation.cc:407] Connection header received
0923 17:27:21.131176 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:21.131184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:21.131289 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:21.131401 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:21.132010 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.133114 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.134713 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.135454 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.135614 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.136893 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:21.136899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:21.136904 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:21.136931 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:21.136986 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:21.137007 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:21.137107 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:21.137316 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 17:27:21.137416 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:27:21.138026 18769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbca88470964e818e0c57e98ca5bc95" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40177 } }
I20260923 17:27:21.138527 18769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:21.138688 18769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bbca88470964e818e0c57e98ca5bc95, State: Initialized, Role: FOLLOWER
I20260923 17:27:21.139410 18769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbca88470964e818e0c57e98ca5bc95" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40177 } }
I20260923 17:27:21.139703 18769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:21.139876 18769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:21.140067 18769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:21.142386 18769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbca88470964e818e0c57e98ca5bc95" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40177 } }
I20260923 17:27:21.142999 18769 leader_election.cc:302] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bbca88470964e818e0c57e98ca5bc95; no voters:
I20260923 17:27:21.143735 18769 leader_election.cc:288] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:21.143844 18779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:21.144563 18779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [term 1 LEADER]: Becoming Leader. State: Replica: 5bbca88470964e818e0c57e98ca5bc95, State: Running, Role: LEADER
I20260923 17:27:21.145215 18779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbca88470964e818e0c57e98ca5bc95" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40177 } }
I20260923 17:27:21.146358 18769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:21.148487 18780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bbca88470964e818e0c57e98ca5bc95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbca88470964e818e0c57e98ca5bc95" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40177 } } }
I20260923 17:27:21.148582 18781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bbca88470964e818e0c57e98ca5bc95. Latest consensus state: current_term: 1 leader_uuid: "5bbca88470964e818e0c57e98ca5bc95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbca88470964e818e0c57e98ca5bc95" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 40177 } } }
I20260923 17:27:21.148924 18780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:21.148983 18781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:21.151650 18785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:21.158722 18785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:21.168023 18785 catalog_manager.cc:1347] Generated new cluster ID: a5955ba55c0d4082b2dffaf41d1a01f3
I20260923 17:27:21.168161 18785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:21.179642 18785 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:21.181651 18785 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:21.198182 18785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bbca88470964e818e0c57e98ca5bc95: Generated new TSK 0
I20260923 17:27:21.199215 18785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:21.210597 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18702
W20260923 17:27:21.212044 18764 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55571
W20260923 17:27:21.212785 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18702: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:27:21.212965 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 18702
2026-09-23T17:27:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1999 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:21 dist-test-slave-j0m4 krb5kdc[18800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:21 dist-test-slave-j0m4 krb5kdc[18800](info): set up 2 sockets
Sep 23 17:27:21 dist-test-slave-j0m4 krb5kdc[18800](info): commencing operation
krb5kdc: starting...
W20260923 17:27:22.391721 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:27:22 dist-test-slave-j0m4 krb5kdc[18800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:27:22.671422 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:45697
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:41609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:45697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:23.017256 18813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:23.017642 18813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:23.017802 18813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:23.025468 18813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:23.025655 18813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:23.025758 18813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:23.025864 18813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:23.025941 18813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:23.036164 18813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:41609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:45697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:45697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:23.037629 18813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:23.038996 18813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:23.051641 18818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:23.051690 18821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:23.051605 18819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:23.057060 18813 server_base.cc:1034] running on GCE node
I20260923 17:27:23.057966 18813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:23.059690 18813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:23.060953 18813 hybrid_clock.cc:648] HybridClock initialized: now 1790184443060909 us; error 45 us; skew 500 ppm
I20260923 17:27:23.061340 18813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:23 dist-test-slave-j0m4 krb5kdc[18800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184443, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:23.064917 18813 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:27:23.066756 18813 webserver.cc:460] Webserver started at http://127.28.72.254:34401/ using document root <none> and password file <none>
I20260923 17:27:23.067667 18813 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:23.067808 18813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:23.068151 18813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:23.071530 18813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "b908f3c9a9944da29c80803e5dcb9510"
format_stamp: "Formatted at 2026-09-23 17:27:23 on dist-test-slave-j0m4"
server_key: "d1bb17725ae211f03c417823707a8e7e"
server_key_iv: "0aaf833dd5033140bc8b887f41459859"
server_key_version: "encryptionkey@0"
I20260923 17:27:23.072392 18813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "b908f3c9a9944da29c80803e5dcb9510"
format_stamp: "Formatted at 2026-09-23 17:27:23 on dist-test-slave-j0m4"
server_key: "d1bb17725ae211f03c417823707a8e7e"
server_key_iv: "0aaf833dd5033140bc8b887f41459859"
server_key_version: "encryptionkey@0"
I20260923 17:27:23.078390 18813 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 17:27:23.082774 18828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:23.084558 18813 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:27:23.084774 18813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "b908f3c9a9944da29c80803e5dcb9510"
format_stamp: "Formatted at 2026-09-23 17:27:23 on dist-test-slave-j0m4"
server_key: "d1bb17725ae211f03c417823707a8e7e"
server_key_iv: "0aaf833dd5033140bc8b887f41459859"
server_key_version: "encryptionkey@0"
I20260923 17:27:23.084977 18813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:23.091323 18813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:23.095773 18813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:23.096107 18813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:23.115989 18813 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:45697
I20260923 17:27:23.116026 18879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:45697 every 8 connection(s)
I20260923 17:27:23.117492 18813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:23.119978 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 18813
I20260923 17:27:23.120182 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:27:23.120788 28963 external_mini_cluster.cc:1428] Setting key fb913d5870c83bda166b52095a50a454
I20260923 17:27:23.122459 18880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:23.129585 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:23.122811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34607 (local address 127.28.72.254:45697)
0923 17:27:23.123841 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:23.123854 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:23.123899 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:27:23.125654 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:23.125678 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:23.126053 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:23.126537 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:23.127745 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:23.127769 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:23.127787 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:23.127803 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:23.127883 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:23.127913 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:23.128174 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:23.128339 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 17:27:23.128556 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":555,"threads_started":1}
I20260923 17:27:23.133684 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:23.136564 18880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510: Bootstrap starting.
I20260923 17:27:23.140969 18880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:23.141182 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:23.138238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58991 (local address 127.28.72.254:45697)
0923 17:27:23.138486 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:23.138493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:23.138935 (+ 442us) server_negotiation.cc:407] Connection header received
0923 17:27:23.139087 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:23.139093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:23.139180 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:23.139296 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:23.140314 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:23.140321 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:23.140325 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:27:23.140346 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:23.140399 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:27:23.140425 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:23.140561 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:23.140647 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 17:27:23.140717 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:27:23.142277 18880 log.cc:824] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:23.145401 18880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510: No bootstrap required, opened a new log
I20260923 17:27:23.153429 18880 raft_consensus.cc:348] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b908f3c9a9944da29c80803e5dcb9510" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45697 } }
I20260923 17:27:23.153880 18880 raft_consensus.cc:374] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:23.153993 18880 raft_consensus.cc:729] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b908f3c9a9944da29c80803e5dcb9510, State: Initialized, Role: FOLLOWER
I20260923 17:27:23.154714 18880 consensus_queue.cc:260] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b908f3c9a9944da29c80803e5dcb9510" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45697 } }
I20260923 17:27:23.155048 18880 raft_consensus.cc:388] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:23.155264 18880 raft_consensus.cc:482] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:23.155462 18880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:23.157708 18880 raft_consensus.cc:504] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b908f3c9a9944da29c80803e5dcb9510" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45697 } }
I20260923 17:27:23.158254 18880 leader_election.cc:302] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b908f3c9a9944da29c80803e5dcb9510; no voters:
I20260923 17:27:23.158910 18880 leader_election.cc:288] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:23.159051 18890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:23.160346 18890 raft_consensus.cc:686] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [term 1 LEADER]: Becoming Leader. State: Replica: b908f3c9a9944da29c80803e5dcb9510, State: Running, Role: LEADER
I20260923 17:27:23.160990 18890 consensus_queue.cc:237] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b908f3c9a9944da29c80803e5dcb9510" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45697 } }
I20260923 17:27:23.161410 18880 sys_catalog.cc:564] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:23.164124 18891 sys_catalog.cc:455] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b908f3c9a9944da29c80803e5dcb9510" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b908f3c9a9944da29c80803e5dcb9510" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45697 } } }
I20260923 17:27:23.164582 18891 sys_catalog.cc:458] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:23.164059 18892 sys_catalog.cc:455] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b908f3c9a9944da29c80803e5dcb9510. Latest consensus state: current_term: 1 leader_uuid: "b908f3c9a9944da29c80803e5dcb9510" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b908f3c9a9944da29c80803e5dcb9510" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 45697 } } }
I20260923 17:27:23.164844 18892 sys_catalog.cc:458] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:23.166795 18898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:23.172672 18898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:23.182232 18898 catalog_manager.cc:1347] Generated new cluster ID: 6c4ca0434e324725991d16e39b450c7d
I20260923 17:27:23.182374 18898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:23.200551 18898 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:23.201151 18898 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:23.216589 18898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b908f3c9a9944da29c80803e5dcb9510: Generated new TSK 0
I20260923 17:27:23.217638 18898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:23.231738 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18813
W20260923 17:27:23.233191 18875 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34607
W20260923 17:27:23.234216 28963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18813: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:27:23.234485 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 18813
2026-09-23T17:27:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2022 ms)
[----------] 18 tests from AuthTokenIssuingTest (36563 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:23 dist-test-slave-j0m4 krb5kdc[18911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:23 dist-test-slave-j0m4 krb5kdc[18911](info): set up 2 sockets
Sep 23 17:27:23 dist-test-slave-j0m4 krb5kdc[18911](info): commencing operation
krb5kdc: starting...
W20260923 17:27:24.416201 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:27:24 dist-test-slave-j0m4 krb5kdc[18911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:27:24.703198 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:35197
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:37109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:35197,127.28.72.253:46875,127.28.72.252:39163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:25.077179 18924 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:27:25.077630 18924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:25.077760 18924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:25.077917 18924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:25.085649 18924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:25.085815 18924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:25.085924 18924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:25.086025 18924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:25.086126 18924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:25.096552 18924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:37109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:35197,127.28.72.253:46875,127.28.72.252:39163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:35197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:25.098165 18924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:25.099598 18924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:25.112232 18929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:25.112159 18930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:25.112197 18932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:25.112847 18924 server_base.cc:1034] running on GCE node
I20260923 17:27:25.113893 18924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:25.115710 18924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:25.116995 18924 hybrid_clock.cc:648] HybridClock initialized: now 1790184445116926 us; error 43 us; skew 500 ppm
I20260923 17:27:25.117390 18924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:25 dist-test-slave-j0m4 krb5kdc[18911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:25.121167 18924 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:27:25.122891 18924 webserver.cc:460] Webserver started at http://127.28.72.254:45589/ using document root <none> and password file <none>
I20260923 17:27:25.123736 18924 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:25.123888 18924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:25.124243 18924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:25.127983 18924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "8b3ee8fa86824918a59e55305395a147"
format_stamp: "Formatted at 2026-09-23 17:27:25 on dist-test-slave-j0m4"
server_key: "b066969263a2d4f5cb44a816032ad3f8"
server_key_iv: "987ea06d3de853e9035a09cca203c754"
server_key_version: "encryptionkey@0"
I20260923 17:27:25.128963 18924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "8b3ee8fa86824918a59e55305395a147"
format_stamp: "Formatted at 2026-09-23 17:27:25 on dist-test-slave-j0m4"
server_key: "b066969263a2d4f5cb44a816032ad3f8"
server_key_iv: "987ea06d3de853e9035a09cca203c754"
server_key_version: "encryptionkey@0"
I20260923 17:27:25.135311 18924 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 17:27:25.139266 18939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:25.140991 18924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:27:25.141247 18924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "8b3ee8fa86824918a59e55305395a147"
format_stamp: "Formatted at 2026-09-23 17:27:25 on dist-test-slave-j0m4"
server_key: "b066969263a2d4f5cb44a816032ad3f8"
server_key_iv: "987ea06d3de853e9035a09cca203c754"
server_key_version: "encryptionkey@0"
I20260923 17:27:25.141526 18924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:25.190851 18924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:25.195808 18924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:25.196130 18924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:25.216274 18924 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:35197
I20260923 17:27:25.216337 18990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:35197 every 8 connection(s)
I20260923 17:27:25.217727 18924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:25.221733 18991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:25.227712 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 18924
I20260923 17:27:25.227950 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:27:25.228752 28963 external_mini_cluster.cc:1428] Setting key 9a4cbcb84988fedfe16e823c2900f9d2
I20260923 17:27:25.229521 18991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:27:25 dist-test-slave-j0m4 krb5kdc[18911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.72.253@KRBTEST.COM; defaulting to no policy
I20260923 17:27:25.258654 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.233713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35197 (local address 127.0.0.1:40142)
0923 17:27:25.234606 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.234658 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:25.236865 (+ 2207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:25.237902 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:25.237908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:25.238940 (+ 1032us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:25.239569 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:25.239603 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.242090 (+ 2487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.242095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:25.243561 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:25.243574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.245015 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.245021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:25.245302 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:25.246137 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:25.246162 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:25.249042 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:25.253859 (+ 4817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:25.253876 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:25.253890 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:25.254567 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:25.255095 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:25.255103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:25.255109 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:25.255370 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:25.256322 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:25.256332 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:25.256606 (+ 274us) client_negotiation.cc:769] Sending connection context
0923 17:27:25.256916 (+ 310us) client_negotiation.cc:240] Negotiation successful
0923 17:27:25.257587 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":86,"thread_start_us":395,"threads_started":1}
I20260923 17:27:25.258708 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.235142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.28.72.254:35197)
0923 17:27:25.235889 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:25.235899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:25.235955 (+ 56us) server_negotiation.cc:407] Connection header received
0923 17:27:25.237272 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:25.237281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:25.237598 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:25.237834 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:25.239891 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.241896 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.243719 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.244720 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.244923 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:25.249342 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:25.249355 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:25.249375 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:25.249418 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:25.253647 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:25.254757 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:25.254766 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:25.254768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:25.254962 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:25.255515 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:25.255519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:25.255521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:25.256145 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:25.256319 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:25.257071 (+ 752us) server_negotiation.cc:299] Negotiation successful
0923 17:27:25.257413 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":519,"thread_start_us":332,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.253@KRBTEST.COM" created.
I20260923 17:27:25.262337 18991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:25.263756 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.263260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:46875 (local address 127.0.0.1:48740)
0923 17:27:25.263460 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.263610 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260923 17:27:25.264082 18941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
I20260923 17:27:25.265811 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.265364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:46875 (local address 127.0.0.1:48746)
0923 17:27:25.265562 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.265639 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 17:27:25.266088 18991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:46875: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.253.keytab.
Entry for principal kudu/127.28.72.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.253.keytab.
I20260923 17:27:25.298395 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.253:46875
--webserver_interface=127.28.72.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.253.keytab
--principal=kudu/127.28.72.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:37109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:35197,127.28.72.253:46875,127.28.72.252:39163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:27:25.315502 18991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } attempt: 1
I20260923 17:27:25.316998 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.316354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:46875 (local address 127.0.0.1:48752)
0923 17:27:25.316616 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.316807 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 17:27:25.318456 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.317931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:46875 (local address 127.0.0.1:48762)
0923 17:27:25.318152 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.318261 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260923 17:27:25.318739 18991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:46875: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
I20260923 17:27:25.387086 18991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } attempt: 2
I20260923 17:27:25.388514 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.387908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:46875 (local address 127.0.0.1:48776)
0923 17:27:25.388208 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.388329 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260923 17:27:25.389693 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.389235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:46875 (local address 127.0.0.1:48790)
0923 17:27:25.389429 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.389519 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 17:27:25.390043 18991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:46875: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
I20260923 17:27:25.481379 18991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } attempt: 3
I20260923 17:27:25.482731 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.482222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:46875 (local address 127.0.0.1:48800)
0923 17:27:25.482458 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.482570 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260923 17:27:25.483916 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.483490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:46875 (local address 127.0.0.1:48806)
0923 17:27:25.483650 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.483737 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 17:27:25.484108 18991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:46875: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
I20260923 17:27:25.627349 18991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } attempt: 4
I20260923 17:27:25.628585 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.627971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:46875 (local address 127.0.0.1:54336)
0923 17:27:25.628294 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.628413 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260923 17:27:25.629638 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.629241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:46875 (local address 127.0.0.1:54350)
0923 17:27:25.629399 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.629492 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 17:27:25.629877 18991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:46875: Network error: Client connection negotiation failed: client connection to 127.28.72.253:46875: connect: Connection refused (error 111)
W20260923 17:27:25.644834 18998 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:27:25.645267 18998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:25.645395 18998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:25.645533 18998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:25.653281 18998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:25.653430 18998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:25.653559 18998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:25.653646 18998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:25.653755 18998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:25.664218 18998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:37109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:35197,127.28.72.253:46875,127.28.72.252:39163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.253.keytab
--principal=kudu/127.28.72.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.253:46875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.72.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:25.665668 18998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:25.667040 18998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:25.679620 19004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:25.679615 19003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:25.679608 19006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:25.680787 18998 server_base.cc:1034] running on GCE node
I20260923 17:27:25.681643 18998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:25.683159 18998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:25.684444 18998 hybrid_clock.cc:648] HybridClock initialized: now 1790184445684383 us; error 31 us; skew 500 ppm
I20260923 17:27:25.684805 18998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:25 dist-test-slave-j0m4 krb5kdc[18911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:25.688177 18998 init.cc:377] Logged in from keytab as kudu/127.28.72.253@KRBTEST.COM (short username kudu)
I20260923 17:27:25.689697 18998 webserver.cc:460] Webserver started at http://127.28.72.253:45883/ using document root <none> and password file <none>
I20260923 17:27:25.690461 18998 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:25.690601 18998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:25.690946 18998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:25.694130 18998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/data/instance:
uuid: "39c61f5f35fa4926a7dad095349b924b"
format_stamp: "Formatted at 2026-09-23 17:27:25 on dist-test-slave-j0m4"
server_key: "e1cc15723a3218d0f1c8e3b9fd409b51"
server_key_iv: "665381ff54029143f9c4bbc803d0df77"
server_key_version: "encryptionkey@0"
I20260923 17:27:25.695042 18998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/wal/instance:
uuid: "39c61f5f35fa4926a7dad095349b924b"
format_stamp: "Formatted at 2026-09-23 17:27:25 on dist-test-slave-j0m4"
server_key: "e1cc15723a3218d0f1c8e3b9fd409b51"
server_key_iv: "665381ff54029143f9c4bbc803d0df77"
server_key_version: "encryptionkey@0"
I20260923 17:27:25.701037 18998 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260923 17:27:25.704957 19013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:25.706594 18998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:27:25.706751 18998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/wal
uuid: "39c61f5f35fa4926a7dad095349b924b"
format_stamp: "Formatted at 2026-09-23 17:27:25 on dist-test-slave-j0m4"
server_key: "e1cc15723a3218d0f1c8e3b9fd409b51"
server_key_iv: "665381ff54029143f9c4bbc803d0df77"
server_key_version: "encryptionkey@0"
I20260923 17:27:25.707002 18998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:25.746654 18998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:25.751304 18998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:25.751536 18998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:25.771801 18998 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.253:46875
I20260923 17:27:25.771822 19064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.253:46875 every 8 connection(s)
I20260923 17:27:25.773303 18998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/data/info.pb
I20260923 17:27:25.777164 19065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:25.777565 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 18998
I20260923 17:27:25.777761 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-1/wal/instance
I20260923 17:27:25.778220 28963 external_mini_cluster.cc:1428] Setting key cbe63f58101832fadbe2c993d76ab17b
I20260923 17:27:25.784041 19065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:27:25 dist-test-slave-j0m4 krb5kdc[18911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.253@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.72.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.252@KRBTEST.COM" created.
I20260923 17:27:25.806161 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.786696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.28.72.254:35197)
0923 17:27:25.787287 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:25.787292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:25.788204 (+ 912us) server_negotiation.cc:407] Connection header received
0923 17:27:25.790278 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:25.790284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:25.790392 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:25.790521 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:25.792352 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.793110 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.794759 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.795273 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.795422 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:25.799613 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:25.799618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:25.799622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:25.799671 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:25.802502 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:25.803463 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:25.803470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:25.803472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:25.803619 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:25.804160 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:25.804166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:25.804169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:25.804564 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:25.804770 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:25.805686 (+ 916us) server_negotiation.cc:299] Negotiation successful
0923 17:27:25.805863 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":338,"threads_started":1}
I20260923 17:27:25.807818 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.787178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35197 (local address 127.0.0.1:50286)
0923 17:27:25.788015 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.788070 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:25.789960 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:25.790799 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:25.790809 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:25.791457 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:25.792092 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:25.792112 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.793299 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.793303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:25.794594 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:25.794607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.795543 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.795551 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:25.795796 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:25.796581 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:25.796614 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:25.799343 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:25.802666 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:25.802676 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:25.802696 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:25.803321 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:25.803740 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:25.803746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:25.803749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:25.803954 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:25.804737 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:25.804747 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:25.805234 (+ 487us) client_negotiation.cc:769] Sending connection context
0923 17:27:25.805599 (+ 365us) client_negotiation.cc:240] Negotiation successful
0923 17:27:25.806210 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":364,"threads_started":1}
I20260923 17:27:25.809780 19065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:27:25 dist-test-slave-j0m4 krb5kdc[18911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.253@KRBTEST.COM for kudu/127.28.72.253@KRBTEST.COM
I20260923 17:27:25.826011 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.810971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:46875 (local address 127.0.0.1:54360)
0923 17:27:25.811239 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.811261 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:25.811386 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:25.812340 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:25.812347 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:25.812908 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:25.813135 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:25.813149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.814677 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.814683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:25.815633 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:25.815645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.816501 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.816506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:25.816752 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:25.817420 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:25.817438 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:25.819787 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:25.823464 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:25.823472 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:25.823478 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:25.823847 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:25.824226 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:25.824231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:25.824235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:25.824374 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:25.824994 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:25.824998 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:25.825141 (+ 143us) client_negotiation.cc:769] Sending connection context
0923 17:27:25.825274 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 17:27:25.825572 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":45}
I20260923 17:27:25.826045 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.811038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.28.72.253:46875)
0923 17:27:25.811866 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:25.811873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:25.811907 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:27:25.811959 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:25.811965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:25.812161 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:25.812355 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:25.813401 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.814523 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.815769 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.816325 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.816492 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:25.820016 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:25.820029 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:25.820043 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:25.820073 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:25.823322 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:25.823996 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:25.824003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:25.824006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:25.824116 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:25.824482 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:25.824494 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:25.824496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:25.824872 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:25.825071 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:25.825435 (+ 364us) server_negotiation.cc:299] Negotiation successful
0923 17:27:25.825686 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":645,"thread_start_us":487,"threads_started":1}
I20260923 17:27:25.828778 19065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:25.829981 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.829500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56022)
0923 17:27:25.829706 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.829811 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260923 17:27:25.830277 19014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
I20260923 17:27:25.831967 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.831539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56036)
0923 17:27:25.831690 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.831825 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 17:27:25.832196 19065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:39163: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.252.keytab.
Entry for principal kudu/127.28.72.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.252.keytab.
I20260923 17:27:25.841121 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.252:39163
--webserver_interface=127.28.72.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.252.keytab
--principal=kudu/127.28.72.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:37109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:35197,127.28.72.253:46875,127.28.72.252:39163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:27:25.881597 19065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } attempt: 1
I20260923 17:27:25.882781 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.882300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56050)
0923 17:27:25.882520 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.882632 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260923 17:27:25.883783 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.883440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56058)
0923 17:27:25.883606 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.883683 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 17:27:25.883987 19065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:39163: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
I20260923 17:27:25.929150 18991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } attempt: 5
Sep 23 17:27:25 dist-test-slave-j0m4 krb5kdc[18911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for kudu/127.28.72.253@KRBTEST.COM
I20260923 17:27:25.945537 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.930161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:46875 (local address 127.0.0.1:54364)
0923 17:27:25.930448 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.930466 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:25.930579 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:25.931029 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:25.931032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:25.931548 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:25.931747 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:25.931757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.933331 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.933336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:25.934187 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:25.934199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.935313 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.935318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:25.935560 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:25.936263 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:25.936282 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:25.938272 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:25.942522 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:25.942527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:25.942530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:25.942995 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:25.943548 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:25.943559 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:25.943562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:25.943709 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:25.944517 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:25.944521 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:25.944668 (+ 147us) client_negotiation.cc:769] Sending connection context
0923 17:27:25.944900 (+ 232us) client_negotiation.cc:240] Negotiation successful
0923 17:27:25.945212 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260923 17:27:25.945720 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.930257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.28.72.253:46875)
0923 17:27:25.930611 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:25.930621 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:25.930651 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:27:25.930733 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:25.930742 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:25.930843 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:25.930975 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:25.931922 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.933125 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.934372 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:25.935156 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:25.935306 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:25.938508 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:25.938516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:25.938522 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:25.938567 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:25.942353 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:25.943170 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:25.943180 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:25.943187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:25.943388 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:25.943843 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:25.943851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:25.943857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:25.944334 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:25.944520 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:25.945033 (+ 513us) server_negotiation.cc:299] Negotiation successful
0923 17:27:25.945288 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260923 17:27:25.946863 18991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:25.947898 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.947508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56064)
0923 17:27:25.947680 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.947766 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 17:27:25.948803 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.948469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56074)
0923 17:27:25.948628 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.948715 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 17:27:25.949028 18991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:39163: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
I20260923 17:27:25.952212 19065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } attempt: 2
I20260923 17:27:25.953555 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.953028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56078)
0923 17:27:25.953245 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.953407 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 17:27:25.954595 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.954249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56086)
0923 17:27:25.954402 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.954481 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 17:27:25.954810 19065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:39163: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
I20260923 17:27:26.000284 18991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } attempt: 1
I20260923 17:27:26.001793 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.001115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56090)
0923 17:27:26.001452 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.001580 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260923 17:27:26.003043 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.002610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56096)
0923 17:27:26.002810 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.002911 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 17:27:26.003357 18991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:39163: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
I20260923 17:27:26.046087 19065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } attempt: 3
I20260923 17:27:26.047457 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.046818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56110)
0923 17:27:26.047078 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.047233 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260923 17:27:26.048709 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.048215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56116)
0923 17:27:26.048484 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.048586 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260923 17:27:26.048933 19065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:39163: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
I20260923 17:27:26.071604 18991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } attempt: 2
I20260923 17:27:26.073004 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.072443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56124)
0923 17:27:26.072721 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.072817 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260923 17:27:26.074431 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.073831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56126)
0923 17:27:26.074099 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.074216 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260923 17:27:26.074787 18991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:39163: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
I20260923 17:27:26.181129 18991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } attempt: 3
I20260923 17:27:26.182374 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.181793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56128)
0923 17:27:26.182051 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.182163 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 17:27:26.183789 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.183309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56142)
0923 17:27:26.183545 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.183635 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260923 17:27:26.184053 18991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:39163: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
I20260923 17:27:26.192166 19065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } attempt: 4
I20260923 17:27:26.193446 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.192946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56148)
0923 17:27:26.193189 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.193277 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 17:27:26.194706 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.194183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56152)
0923 17:27:26.194412 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.194533 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260923 17:27:26.195029 19065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:39163: Network error: Client connection negotiation failed: client connection to 127.28.72.252:39163: connect: Connection refused (error 111)
W20260923 17:27:26.202105 19073 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:27:26.202462 19073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:26.202527 19073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:26.202677 19073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:26.210580 19073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:26.210724 19073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:26.210839 19073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:26.210929 19073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:26.211041 19073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:26.224967 19073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:37109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:35197,127.28.72.253:46875,127.28.72.252:39163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.252.keytab
--principal=kudu/127.28.72.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.252:39163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.72.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:26.226560 19073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:26.228196 19073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:26.240216 19081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:26.240216 19079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:26.240420 19078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:26.241076 19073 server_base.cc:1034] running on GCE node
I20260923 17:27:26.241906 19073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:26.243481 19073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:26.244720 19073 hybrid_clock.cc:648] HybridClock initialized: now 1790184446244636 us; error 56 us; skew 500 ppm
I20260923 17:27:26.245091 19073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:26 dist-test-slave-j0m4 krb5kdc[18911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184446, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:26.254576 19073 init.cc:377] Logged in from keytab as kudu/127.28.72.252@KRBTEST.COM (short username kudu)
I20260923 17:27:26.256312 19073 webserver.cc:460] Webserver started at http://127.28.72.252:36127/ using document root <none> and password file <none>
I20260923 17:27:26.257100 19073 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:26.257261 19073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:26.257609 19073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:26.260811 19073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/data/instance:
uuid: "60f41f1a9cda4fccaf52479239893d5f"
format_stamp: "Formatted at 2026-09-23 17:27:26 on dist-test-slave-j0m4"
server_key: "d17fc03da01e31e9d24d5340524a9363"
server_key_iv: "c8ffae6cba0ec68de800f6a9f3c39864"
server_key_version: "encryptionkey@0"
I20260923 17:27:26.261662 19073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/wal/instance:
uuid: "60f41f1a9cda4fccaf52479239893d5f"
format_stamp: "Formatted at 2026-09-23 17:27:26 on dist-test-slave-j0m4"
server_key: "d17fc03da01e31e9d24d5340524a9363"
server_key_iv: "c8ffae6cba0ec68de800f6a9f3c39864"
server_key_version: "encryptionkey@0"
I20260923 17:27:26.267488 19073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:27:26.271386 19088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:26.272993 19073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:27:26.273222 19073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/wal
uuid: "60f41f1a9cda4fccaf52479239893d5f"
format_stamp: "Formatted at 2026-09-23 17:27:26 on dist-test-slave-j0m4"
server_key: "d17fc03da01e31e9d24d5340524a9363"
server_key_iv: "c8ffae6cba0ec68de800f6a9f3c39864"
server_key_version: "encryptionkey@0"
I20260923 17:27:26.273524 19073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:26.304883 19073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:26.309693 19073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:26.309994 19073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:26.329519 19073 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.252:39163
I20260923 17:27:26.329537 19139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.252:39163 every 8 connection(s)
I20260923 17:27:26.330924 19073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/data/info.pb
I20260923 17:27:26.334808 19140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:26.340339 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 19073
I20260923 17:27:26.340547 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184070511153-28963-0/minicluster-data/master-2/wal/instance
I20260923 17:27:26.341027 28963 external_mini_cluster.cc:1428] Setting key fb55ea178a341bc3f867796a7860b949
I20260923 17:27:26.341984 19140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:27:26 dist-test-slave-j0m4 krb5kdc[18911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184446, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.252@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
Sep 23 17:27:26 dist-test-slave-j0m4 krb5kdc[18911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.252@KRBTEST.COM
I20260923 17:27:26.361299 18991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } attempt: 4
I20260923 17:27:26.367223 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.346217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.28.72.254:35197)
0923 17:27:26.347755 (+ 1538us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.347761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.347779 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:27:26.347856 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.347863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.347951 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.348061 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.349940 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.351290 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.353624 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.354356 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.354510 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.359296 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:26.359307 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:26.359410 (+ 103us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:26.359475 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:26.363288 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.364338 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.364346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.364352 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.364533 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.365163 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.365172 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.365177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.365538 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:26.365703 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:26.366545 (+ 842us) server_negotiation.cc:299] Negotiation successful
0923 17:27:26.366786 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1412,"thread_start_us":1290,"threads_started":1}
I20260923 17:27:26.368559 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.345691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:35197 (local address 127.0.0.1:50300)
0923 17:27:26.346589 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.346637 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:26.347171 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:26.348188 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:26.348198 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:26.349125 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:26.349702 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:26.349732 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.351474 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.351478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:26.353409 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:26.353424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.354603 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.354608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:26.354841 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.355825 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:26.355858 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:26.359002 (+ 3144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:26.363436 (+ 4434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:26.363449 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:26.363464 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:26.364198 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:26.364658 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:26.364664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:26.364667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:26.365022 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:26.365693 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:26.365701 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:26.366114 (+ 413us) client_negotiation.cc:769] Sending connection context
0923 17:27:26.366416 (+ 302us) client_negotiation.cc:240] Negotiation successful
0923 17:27:26.366954 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":49,"thread_start_us":452,"threads_started":1}
I20260923 17:27:26.370183 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.343412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.28.72.252:39163)
0923 17:27:26.344219 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.344238 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.344301 (+ 63us) server_negotiation.cc:407] Connection header received
0923 17:27:26.347171 (+ 2870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.347200 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.347536 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.347747 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.349739 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.351756 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.353611 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.354690 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.354893 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.360147 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:26.360156 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:26.360171 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:26.360203 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:26.364526 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.365523 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.365540 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.365542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.365719 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.367443 (+ 1724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.367448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.367452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.367965 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:26.369269 (+ 1304us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:26.369464 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 17:27:26.369763 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":574,"thread_start_us":355,"threads_started":1}
I20260923 17:27:26.370652 19140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:27:26 dist-test-slave-j0m4 krb5kdc[18911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for kudu/127.28.72.252@KRBTEST.COM
I20260923 17:27:26.379746 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.362260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56162)
0923 17:27:26.362571 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.362592 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:26.362710 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:26.363679 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:26.363684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:26.364098 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:26.364257 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:26.364295 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.365823 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.365832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:26.366802 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:26.366814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.368004 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.368010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:26.368224 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.369054 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:26.369082 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:26.371974 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:26.376175 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:26.376183 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:26.376187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:26.376833 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:26.377288 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:26.377293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:26.377297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:26.377496 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:26.378588 (+ 1092us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:26.378594 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:26.378785 (+ 191us) client_negotiation.cc:769] Sending connection context
0923 17:27:26.378955 (+ 170us) client_negotiation.cc:240] Negotiation successful
0923 17:27:26.379306 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
Sep 23 17:27:26 dist-test-slave-j0m4 krb5kdc[18911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184446, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.252@KRBTEST.COM for kudu/127.28.72.253@KRBTEST.COM
I20260923 17:27:26.380079 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.362083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.28.72.252:39163)
0923 17:27:26.363356 (+ 1273us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.363363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.363387 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:27:26.363454 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.363463 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.363524 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.363656 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.364428 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.365721 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.366957 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.367828 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.367977 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.372210 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:26.372217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:26.372221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:26.372254 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:26.375969 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.376962 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.376968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.376972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.377130 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.377971 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.377976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.377980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.378394 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:26.378611 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:26.379073 (+ 462us) server_negotiation.cc:299] Negotiation successful
0923 17:27:26.379412 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1127,"thread_start_us":856,"threads_started":1}
Sep 23 17:27:26 dist-test-slave-j0m4 krb5kdc[18911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:27:26.385955 18991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147: Bootstrap starting.
I20260923 17:27:26.388053 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.371556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:46875 (local address 127.0.0.1:54380)
0923 17:27:26.371749 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.371770 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:26.371873 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:26.372419 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:26.372425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:26.372929 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:26.373168 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:26.373179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.374678 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.374682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:26.375912 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:26.375924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.377090 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.377095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:26.377333 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.377971 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:26.377994 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:26.380896 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:26.385041 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:26.385048 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:26.385052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:26.385539 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:26.385952 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:26.385958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:26.385961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:26.386153 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:26.386973 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:26.386981 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:26.387201 (+ 220us) client_negotiation.cc:769] Sending connection context
0923 17:27:26.387378 (+ 177us) client_negotiation.cc:240] Negotiation successful
0923 17:27:26.387677 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 17:27:26.388082 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.371851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.28.72.253:46875)
0923 17:27:26.372057 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.372064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.372083 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:27:26.372153 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.372157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.372265 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.372371 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.373389 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.374517 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.376066 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.376735 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.376877 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.381119 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:26.381125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:26.381128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:26.381166 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:26.384872 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.385671 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.385677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.385681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.385820 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.386293 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.386298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.386301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.386726 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:26.386850 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:26.387492 (+ 642us) server_negotiation.cc:299] Negotiation successful
0923 17:27:26.387704 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:27:26.389349 19140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:26.390044 18991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:26.391624 18991 log.cc:824] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:26.394338 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.374795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50310 (local address 127.28.72.254:35197)
0923 17:27:26.375079 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.375085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.375105 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:27:26.375189 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.375194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.375281 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.375385 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.376805 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.377773 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.380010 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.380643 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.381425 (+ 782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.386374 (+ 4949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:26.386382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:26.386388 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:26.386430 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:26.391166 (+ 4736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.392038 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.392043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.392046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.392224 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.392711 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.392717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.392720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.393074 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:26.393391 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:26.393745 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 17:27:26.393998 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 17:27:26.395376 18991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147: No bootstrap required, opened a new log
Sep 23 17:27:26 dist-test-slave-j0m4 krb5kdc[18911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184446, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.252@KRBTEST.COM for kudu/127.28.72.252@KRBTEST.COM
I20260923 17:27:26.405179 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.390129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56170)
0923 17:27:26.390381 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.390399 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:26.390504 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:26.390812 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:26.390816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:26.391336 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:26.391546 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:26.391558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.392920 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.392928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:26.394293 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:26.394331 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.395027 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.395033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:26.395764 (+ 731us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.396769 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:26.396791 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:26.398950 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:26.402112 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:26.402118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:26.402123 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:26.402637 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:26.403075 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:26.403078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:26.403080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:26.403279 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:26.403981 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:26.403987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:26.404174 (+ 187us) client_negotiation.cc:769] Sending connection context
0923 17:27:26.404353 (+ 179us) client_negotiation.cc:240] Negotiation successful
0923 17:27:26.404721 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260923 17:27:26.405264 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.390241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56170 (local address 127.28.72.252:39163)
0923 17:27:26.390384 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.390390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.390453 (+ 63us) server_negotiation.cc:407] Connection header received
0923 17:27:26.390620 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.390624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.390696 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.390797 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.391700 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.392772 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.394448 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.394878 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.395020 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.399185 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:26.399191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:26.399197 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:26.399236 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:26.401965 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.402765 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.402772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.402775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.402937 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.403410 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.403416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.403420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.403836 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:26.403998 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:26.404521 (+ 523us) server_negotiation.cc:299] Negotiation successful
0923 17:27:26.404897 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":45}
Sep 23 17:27:26 dist-test-slave-j0m4 krb5kdc[18911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.253@KRBTEST.COM
I20260923 17:27:26.406782 18991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } }
I20260923 17:27:26.407258 18991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3ee8fa86824918a59e55305395a147, State: Initialized, Role: FOLLOWER
I20260923 17:27:26.407861 18991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } }
I20260923 17:27:26.409206 19148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } } }
I20260923 17:27:26.409647 19148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:26.410447 18991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:26.410998 19140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f: Bootstrap starting.
I20260923 17:27:26.415241 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.396610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.28.72.253:46875)
0923 17:27:26.396805 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.396811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.396831 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:27:26.396958 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.396964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.397040 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.397144 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.398771 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.399947 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.401778 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.402336 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.402489 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.408215 (+ 5726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:26.408222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:26.408227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:26.408273 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:26.411894 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.412811 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.412818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.412823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.412998 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.413535 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.413539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.413543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.413891 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:26.414185 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:26.414571 (+ 386us) server_negotiation.cc:299] Negotiation successful
0923 17:27:26.414810 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:27:26.415057 19140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:26.416621 19140 log.cc:824] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:26.420029 19140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f: No bootstrap required, opened a new log
I20260923 17:27:26.421280 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:26.425161 18966 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8b3ee8fa86824918a59e55305395a147"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50310
I20260923 17:27:26.425751 18966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 17:27:26.425983 18966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:26.428694 18966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } }
I20260923 17:27:26.429797 19140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } }
I20260923 17:27:26.430177 19140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f41f1a9cda4fccaf52479239893d5f, State: Initialized, Role: FOLLOWER
I20260923 17:27:26.430770 19140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } }
I20260923 17:27:26.431545 18966 leader_election.cc:288] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [CANDIDATE]: Term 1 election: Requested vote from peers 39c61f5f35fa4926a7dad095349b924b (127.28.72.253:46875), 60f41f1a9cda4fccaf52479239893d5f (127.28.72.252:39163)
I20260923 17:27:26.433604 19115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b3ee8fa86824918a59e55305395a147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "60f41f1a9cda4fccaf52479239893d5f"
I20260923 17:27:26.434929 19115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:26.432209 19161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } } }
I20260923 17:27:26.439327 19115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b3ee8fa86824918a59e55305395a147 in term 1.
I20260923 17:27:26.439623 19161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:26.440117 18940 leader_election.cc:302] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60f41f1a9cda4fccaf52479239893d5f, 8b3ee8fa86824918a59e55305395a147; no voters:
I20260923 17:27:26.440795 19148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:26.441720 19148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [term 1 LEADER]: Becoming Leader. State: Replica: 8b3ee8fa86824918a59e55305395a147, State: Running, Role: LEADER
I20260923 17:27:26.442482 19148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } }
I20260923 17:27:26.446826 19164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b3ee8fa86824918a59e55305395a147. Latest consensus state: current_term: 1 leader_uuid: "8b3ee8fa86824918a59e55305395a147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } } }
I20260923 17:27:26.447578 19164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [sys.catalog]: This master's current role is: LEADER
W20260923 17:27:26.449191 18945 tablet.cc:2369] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:27:26.433504 19140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:26.460660 19168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:26.463780 19160 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:27:26.467589 19168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:26.485229 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.456330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56174 (local address 127.28.72.252:39163)
0923 17:27:26.456530 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.456536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.465723 (+ 9187us) server_negotiation.cc:407] Connection header received
0923 17:27:26.466235 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.466241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.466333 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.466552 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.469019 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.470069 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.472113 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.472862 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.472947 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.477224 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:26.477232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:26.477237 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:26.477298 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:26.481442 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.482395 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.482402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.482407 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.482606 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.483159 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.483165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.483168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.483577 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:26.483884 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:26.484550 (+ 666us) server_negotiation.cc:299] Negotiation successful
0923 17:27:26.484799 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:27:26.489411 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.453920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.28.72.254:35197)
0923 17:27:26.454157 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.454167 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.455821 (+ 1654us) server_negotiation.cc:407] Connection header received
0923 17:27:26.457038 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.457049 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.457162 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.457252 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.466734 (+ 9482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.468068 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.471978 (+ 3910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.472765 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.472877 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.481993 (+ 9116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:26.482002 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:26.482008 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:26.482088 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:26.485938 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.486781 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.486788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.486792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.486970 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.487455 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.487460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.487463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.487860 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:26.488132 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:26.488580 (+ 448us) server_negotiation.cc:299] Negotiation successful
0923 17:27:26.488798 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 17:27:26.494380 19065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } attempt: 5
I20260923 17:27:26.497593 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.453220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.28.72.253:46875)
0923 17:27:26.453416 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.453423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.460877 (+ 7454us) server_negotiation.cc:407] Connection header received
0923 17:27:26.463352 (+ 2475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.463363 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.463502 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.463598 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.470104 (+ 6506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.471247 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.477825 (+ 6578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.478613 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.478751 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.489217 (+ 10466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:26.489227 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:26.489235 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:26.489285 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:26.493231 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.494373 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.494379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.494382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.494581 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.495181 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.495190 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.495196 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.495701 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:26.496022 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:26.496715 (+ 693us) server_negotiation.cc:299] Negotiation successful
0923 17:27:26.497074 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 23 17:27:26 dist-test-slave-j0m4 krb5kdc[18911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.253@KRBTEST.COM for kudu/127.28.72.252@KRBTEST.COM
I20260923 17:27:26.468492 18924 master_runner.cc:186] Error getting master registration for 127.28.72.253:46875: OK, instance_id { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" instance_seqno: 1790184445765994 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:27:26.508067 18924 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:27:26.517812 19115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f [term 1 FOLLOWER]: Refusing update from remote peer 8b3ee8fa86824918a59e55305395a147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:27:26.518008 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.495723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.28.72.252:39163)
0923 17:27:26.495901 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.495907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.495925 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:27:26.495982 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.495987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.496059 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.496151 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.497299 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.498463 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.500393 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.501071 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.501208 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.506042 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:26.506052 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:26.506057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:26.506099 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:26.511073 (+ 4974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.512213 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.512222 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.512227 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.512405 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:26.513261 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:26.513271 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:26.513278 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:26.514210 (+ 932us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:26.514505 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:26.515636 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0923 17:27:26.516223 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:27:26.519181 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.494975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:39163 (local address 127.0.0.1:56182)
0923 17:27:26.495382 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.495406 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:26.495521 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:26.496384 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:26.496390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:26.496876 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:26.497086 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:26.497098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.498659 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.498665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:26.500167 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:26.500183 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.501260 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.501270 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:26.501633 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.502411 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:26.502444 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:26.505787 (+ 3343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:26.511340 (+ 5553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:26.511345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:26.511349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:26.512045 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:26.512581 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:26.512590 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:26.512596 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:26.512990 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:26.514694 (+ 1704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:26.514702 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:26.515070 (+ 368us) client_negotiation.cc:769] Sending connection context
0923 17:27:26.516352 (+ 1282us) client_negotiation.cc:240] Negotiation successful
0923 17:27:26.517945 (+ 1593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260923 17:27:26.519364 19164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:27:26.529232 19065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b: Bootstrap starting.
I20260923 17:27:26.533717 19065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:26.535280 19065 log.cc:824] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:26.537786 19161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b3ee8fa86824918a59e55305395a147. Latest consensus state: current_term: 1 leader_uuid: "8b3ee8fa86824918a59e55305395a147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } } }
I20260923 17:27:26.538118 19161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:26.541527 19065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b: No bootstrap required, opened a new log
I20260923 17:27:26.534180 19073 master_runner.cc:186] Error getting master registration for 127.28.72.253:46875: OK, instance_id { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" instance_seqno: 1790184445765994 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:27:26.545181 19073 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:27:26.550661 19183 mvcc.cc:205] Tried to move back new op lower bound from 7332595492908482560 to 7332595492635762688. Current Snapshot: MvccSnapshot[applied={T|T < 7332595492908482560}]
I20260923 17:27:26.552563 19164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b3ee8fa86824918a59e55305395a147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } } }
I20260923 17:27:26.552855 19164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:26.556471 19168 catalog_manager.cc:1347] Generated new cluster ID: d50030be71d6466d99674d2bea99b508
I20260923 17:27:26.556576 19168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:26.556681 19065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } }
I20260923 17:27:26.557158 19065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c61f5f35fa4926a7dad095349b924b, State: Initialized, Role: FOLLOWER
I20260923 17:27:26.557790 19065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } }
I20260923 17:27:26.558961 19161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b3ee8fa86824918a59e55305395a147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } } }
I20260923 17:27:26.559223 19161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:26.565338 19191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } } }
I20260923 17:27:26.571386 19191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:26.568595 19039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b [term 0 FOLLOWER]: Advancing to term 1
W20260923 17:27:26.571542 19184 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260923 17:27:26.568796 19040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b3ee8fa86824918a59e55305395a147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "39c61f5f35fa4926a7dad095349b924b"
W20260923 17:27:26.575666 19184 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:27:26.575944 19039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b [term 1 FOLLOWER]: Refusing update from remote peer 8b3ee8fa86824918a59e55305395a147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:27:26.578979 19040 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b3ee8fa86824918a59e55305395a147 in term 1.
I20260923 17:27:26.568153 19065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:26.581530 19164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:27:26.587605 19168 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:26.589658 19168 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:26.620051 19191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b3ee8fa86824918a59e55305395a147. Latest consensus state: current_term: 1 leader_uuid: "8b3ee8fa86824918a59e55305395a147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } } }
I20260923 17:27:26.620404 19191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:26.620551 19148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b3ee8fa86824918a59e55305395a147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } } }
I20260923 17:27:26.620870 19148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:26.620810 19196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b3ee8fa86824918a59e55305395a147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3ee8fa86824918a59e55305395a147" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 35197 } } peers { permanent_uuid: "39c61f5f35fa4926a7dad095349b924b" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 46875 } } peers { permanent_uuid: "60f41f1a9cda4fccaf52479239893d5f" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 39163 } } }
I20260923 17:27:26.621209 19196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:26.627298 19168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b3ee8fa86824918a59e55305395a147: Generated new TSK 0
I20260923 17:27:26.628871 19168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:26.730674 19209 catalog_manager.cc:1259] Loaded cluster ID: d50030be71d6466d99674d2bea99b508
I20260923 17:27:26.730806 19209 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b: loading cluster ID for follower catalog manager: success
I20260923 17:27:26.734974 19209 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b: acquiring CA information for follower catalog manager: success
I20260923 17:27:26.737615 19209 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 39c61f5f35fa4926a7dad095349b924b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 17:27:26.772285 19145 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:26.757481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.28.72.254:35197)
0923 17:27:26.757707 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.757713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.768718 (+ 11005us) server_negotiation.cc:407] Connection header received
0923 17:27:26.770515 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.770526 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.770616 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.770694 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.772019 (+ 1325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50320: BlockingRecv error: recv got EOF from 127.0.0.1:50320 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260923 17:27:26.776129 19070 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:26.764649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.28.72.253:46875)
0923 17:27:26.764918 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.764927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.773873 (+ 8946us) server_negotiation.cc:407] Connection header received
0923 17:27:26.774698 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.774704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.774793 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.774863 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.775864 (+ 1001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54408: BlockingRecv error: recv got EOF from 127.0.0.1:54408 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260923 17:27:26.780601 19146 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:26.769463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56192 (local address 127.28.72.252:39163)
0923 17:27:26.769701 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.769710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.777441 (+ 7731us) server_negotiation.cc:407] Connection header received
0923 17:27:26.778098 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.778103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.778192 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.778269 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.780339 (+ 2070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56192: BlockingRecv error: recv got EOF from 127.0.0.1:56192 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 17:27:26.856421 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.833195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.28.72.253:46875)
0923 17:27:26.833396 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.833402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.847784 (+ 14382us) server_negotiation.cc:407] Connection header received
0923 17:27:26.848058 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.848065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.848321 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.848520 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:26.849248 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.850486 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.852770 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.853491 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.853650 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.854496 (+ 846us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:26.855260 (+ 764us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:26.855400 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:26.855948 (+ 548us) server_negotiation.cc:299] Negotiation successful
0923 17:27:26.856049 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:27:26.921027 19146 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:26.907653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.28.72.252:39163)
0923 17:27:26.907859 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.907865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.919011 (+ 11146us) server_negotiation.cc:407] Connection header received
0923 17:27:26.919357 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.919363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.919500 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.919584 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:26.920748 (+ 1164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56208: BlockingRecv error: recv got EOF from 127.0.0.1:56208 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:27:26.972693 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.954755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.28.72.253:46875)
0923 17:27:26.954974 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:26.954980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:26.965589 (+ 10609us) server_negotiation.cc:407] Connection header received
0923 17:27:26.966123 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:26.966131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:26.966223 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:26.966288 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:26.966983 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.968170 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.970302 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:26.970991 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:26.971089 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:26.971765 (+ 676us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:26.971960 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:26.972024 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:26.972270 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 17:27:26.972369 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260923 17:27:27.023854 19146 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:27.010515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.28.72.252:39163)
0923 17:27:27.011052 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.011059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.021559 (+ 10500us) server_negotiation.cc:407] Connection header received
0923 17:27:27.022129 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.022137 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.022251 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.022344 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:27.023567 (+ 1223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56218: BlockingRecv error: recv got EOF from 127.0.0.1:56218 (error 108)
Metrics: {"server-negotiator.queue_time_us":390}
I20260923 17:27:27.078440 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.060369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.28.72.253:46875)
0923 17:27:27.060560 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.060567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.071248 (+ 10681us) server_negotiation.cc:407] Connection header received
0923 17:27:27.071820 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.071830 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.071936 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.072006 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:27.072704 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.073896 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.076048 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.076766 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.076880 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:27.077505 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:27.077684 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:27.077749 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:27.077988 (+ 239us) server_negotiation.cc:299] Negotiation successful
0923 17:27:27.078086 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 17:27:27.127103 19146 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:27.113129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.28.72.252:39163)
0923 17:27:27.113314 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.113321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.125181 (+ 11860us) server_negotiation.cc:407] Connection header received
0923 17:27:27.125530 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.125536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.125633 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.125715 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:27.126827 (+ 1112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56224: BlockingRecv error: recv got EOF from 127.0.0.1:56224 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:27:27.155499 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.148124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.28.72.253:46875)
0923 17:27:27.148333 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.148339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.148940 (+ 601us) server_negotiation.cc:407] Connection header received
0923 17:27:27.149158 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.149165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.149268 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.149385 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:27.149863 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.151072 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.153147 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.153903 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.154070 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:27.154613 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:27.154769 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:27.154866 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:27.155043 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 17:27:27.155175 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260923 17:27:27.165019 19146 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:27.162479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.28.72.252:39163)
0923 17:27:27.162730 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.162736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.163277 (+ 541us) server_negotiation.cc:407] Connection header received
0923 17:27:27.163562 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.163569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.163675 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.163789 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:27.164758 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56238: BlockingRecv error: recv got EOF from 127.0.0.1:56238 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
I20260923 17:27:27.183938 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.176607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.28.72.253:46875)
0923 17:27:27.176857 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.176865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.177378 (+ 513us) server_negotiation.cc:407] Connection header received
0923 17:27:27.177586 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.177592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.177680 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.177784 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:27.178277 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.179567 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.181665 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.182404 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.182523 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:27.183070 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:27.183237 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:27.183326 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:27.183516 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 17:27:27.183611 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260923 17:27:27.195828 19146 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:27.193395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.28.72.252:39163)
0923 17:27:27.193596 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.193604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.194193 (+ 589us) server_negotiation.cc:407] Connection header received
0923 17:27:27.194416 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.194421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.194512 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.194615 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:27.195575 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56250: BlockingRecv error: recv got EOF from 127.0.0.1:56250 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:27:27.223680 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.216354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.28.72.253:46875)
0923 17:27:27.216564 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.216571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.217188 (+ 617us) server_negotiation.cc:407] Connection header received
0923 17:27:27.217419 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.217424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.217513 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.217610 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:27.218162 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.219437 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.221462 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.222190 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.222339 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:27.222832 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:27.222973 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:27.223035 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:27.223300 (+ 265us) server_negotiation.cc:299] Negotiation successful
0923 17:27:27.223394 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260923 17:27:27.233548 19146 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:27.230866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.28.72.252:39163)
0923 17:27:27.231143 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.231150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.231713 (+ 563us) server_negotiation.cc:407] Connection header received
0923 17:27:27.232036 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.232042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.232139 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.232222 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:27.233275 (+ 1053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56256: BlockingRecv error: recv got EOF from 127.0.0.1:56256 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:27:27.276988 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.270488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54454 (local address 127.28.72.253:46875)
0923 17:27:27.270745 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.270752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.271153 (+ 401us) server_negotiation.cc:407] Connection header received
0923 17:27:27.271385 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.271391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.271484 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.271631 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:27.272130 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.273284 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.274756 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.275566 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.275697 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:27.276143 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:27.276269 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:27.276379 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:27.276546 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 17:27:27.276635 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260923 17:27:27.293735 19146 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:27.291668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.28.72.252:39163)
0923 17:27:27.291897 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.291904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.292315 (+ 411us) server_negotiation.cc:407] Connection header received
0923 17:27:27.292491 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.292496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.292589 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.292713 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:27.293517 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56270: BlockingRecv error: recv got EOF from 127.0.0.1:56270 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:27:27.367268 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.361346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.28.72.253:46875)
0923 17:27:27.361547 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.361551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.362118 (+ 567us) server_negotiation.cc:407] Connection header received
0923 17:27:27.362277 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.362281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.362421 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.362504 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:27.362991 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.363912 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.365293 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.365744 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.365852 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:27.366336 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:27.366529 (+ 193us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:27.366604 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:27.366788 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 17:27:27.366860 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 17:27:27.376008 19146 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:27.373802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.28.72.252:39163)
0923 17:27:27.374052 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.374060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.374420 (+ 360us) server_negotiation.cc:407] Connection header received
0923 17:27:27.374651 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.374657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.374744 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.374863 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:27.375736 (+ 873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56282: BlockingRecv error: recv got EOF from 127.0.0.1:56282 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:27:27.517597 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.512028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54470 (local address 127.28.72.253:46875)
0923 17:27:27.512246 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.512250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.512626 (+ 376us) server_negotiation.cc:407] Connection header received
0923 17:27:27.512845 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.512850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.512943 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.513037 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:27.513590 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.514397 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.515939 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.516407 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.516520 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:27.516937 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:27.517055 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:27.517132 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:27.517312 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 17:27:27.517388 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 17:27:27.526214 19146 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:27.524209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.28.72.252:39163)
0923 17:27:27.524583 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.524590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.524917 (+ 327us) server_negotiation.cc:407] Connection header received
0923 17:27:27.525136 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.525140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.525210 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.525323 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:27.526015 (+ 692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56286: BlockingRecv error: recv got EOF from 127.0.0.1:56286 (error 108)
Metrics: {"server-negotiator.queue_time_us":197}
I20260923 17:27:27.577819 19184 catalog_manager.cc:1259] Loaded cluster ID: d50030be71d6466d99674d2bea99b508
I20260923 17:27:27.577950 19184 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f: loading cluster ID for follower catalog manager: success
I20260923 17:27:27.581167 19184 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f: acquiring CA information for follower catalog manager: success
I20260923 17:27:27.583343 19184 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60f41f1a9cda4fccaf52479239893d5f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:27:27.791812 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.786021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.28.72.253:46875)
0923 17:27:27.786282 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.786289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.786670 (+ 381us) server_negotiation.cc:407] Connection header received
0923 17:27:27.786941 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.786946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.787020 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.787152 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:27.787596 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.788393 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.789925 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.790387 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.790536 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:27.790959 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:27.791086 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:27.791220 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:27.791399 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 17:27:27.791487 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260923 17:27:27.804072 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.797841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.28.72.252:39163)
0923 17:27:27.798111 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.798118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.798481 (+ 363us) server_negotiation.cc:407] Connection header received
0923 17:27:27.798731 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.798744 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.798912 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.799042 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:27.799577 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.800338 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.801681 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.802096 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.802238 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:27.802778 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:27.803360 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:27.803529 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:27.803646 (+ 117us) server_negotiation.cc:299] Negotiation successful
0923 17:27:27.803747 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260923 17:27:27.819046 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.811326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.28.72.252:39163)
0923 17:27:27.811589 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.811598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.813318 (+ 1720us) server_negotiation.cc:407] Connection header received
0923 17:27:27.813583 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.813589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.813667 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.813767 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:27.814344 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.815224 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.816955 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.817525 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.817652 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:27.818120 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:27.818286 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:27.818376 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:27.818612 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 17:27:27.818704 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 17:27:27.819473 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.810762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54502 (local address 127.28.72.253:46875)
0923 17:27:27.811032 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.811041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.812571 (+ 1530us) server_negotiation.cc:407] Connection header received
0923 17:27:27.812874 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.812883 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.812987 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.813090 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:27.813790 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.814973 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.817392 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.818057 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.818194 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:27.818677 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:27.818804 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:27.818873 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:27.819059 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 17:27:27.819165 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260923 17:27:27.820756 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.810445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50324 (local address 127.28.72.254:35197)
0923 17:27:27.812644 (+ 2199us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:27.812650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:27.812667 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:27:27.812806 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:27.812814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:27.813071 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:27.813297 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:27.814035 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.815247 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.817634 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:27.818262 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:27.818426 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:27.819252 (+ 826us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:27.819830 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:27.819995 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:27.820154 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 17:27:27.820556 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2078,"thread_start_us":1980,"threads_started":1}
I20260923 17:27:27.826402 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18924
I20260923 17:27:27.827598 18986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:27:27.942924 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 18924
I20260923 17:27:27.964500 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 18998
I20260923 17:27:27.965925 19060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:27:28.076160 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 18998
I20260923 17:27:28.097626 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 19073
I20260923 17:27:28.098829 19135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:27:28.208408 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 19073
2026-09-23T17:27:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4977 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:28 dist-test-slave-j0m4 krb5kdc[19331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:28 dist-test-slave-j0m4 krb5kdc[19331](info): set up 2 sockets
Sep 23 17:27:28 dist-test-slave-j0m4 krb5kdc[19331](info): commencing operation
krb5kdc: starting...
W20260923 17:27:29.340029 28963 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/test-user.keytab.
Sep 23 17:27:29 dist-test-slave-j0m4 krb5kdc[19331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.72.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
Entry for principal kudu/127.28.72.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab.
I20260923 17:27:29.626518 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:37297
--webserver_interface=127.28.72.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:37297,127.28.72.253:34333,127.28.72.252:43221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:29.977067 19344 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:27:29.977427 19344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:29.977497 19344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:29.977583 19344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:29.985126 19344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:29.985236 19344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:29.985306 19344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:29.985358 19344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:29.985414 19344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:29.995467 19344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:37297,127.28.72.253:34333,127.28.72.252:43221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.254.keytab
--principal=kudu/127.28.72.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.254:37297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.72.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:29.996789 19344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:29.998114 19344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:30.011332 19349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:30.011387 19350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:30.011332 19352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:30.012810 19344 server_base.cc:1034] running on GCE node
I20260923 17:27:30.013602 19344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:30.015216 19344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:30.016510 19344 hybrid_clock.cc:648] HybridClock initialized: now 1790184450016428 us; error 77 us; skew 500 ppm
I20260923 17:27:30.016904 19344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:30 dist-test-slave-j0m4 krb5kdc[19331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184450, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:30.020469 19344 init.cc:377] Logged in from keytab as kudu/127.28.72.254@KRBTEST.COM (short username kudu)
I20260923 17:27:30.022291 19344 webserver.cc:460] Webserver started at http://127.28.72.254:41077/ using document root <none> and password file <none>
I20260923 17:27:30.023208 19344 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:30.023362 19344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:30.023734 19344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:30.027024 19344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/data/instance:
uuid: "085c41158d3040fab5c0803a10952766"
format_stamp: "Formatted at 2026-09-23 17:27:30 on dist-test-slave-j0m4"
server_key: "4791348567cb8b8a50ef4b7e59a838eb"
server_key_iv: "28d2fedb047d1180fa9d26647b02c68c"
server_key_version: "encryptionkey@0"
I20260923 17:27:30.027983 19344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/wal/instance:
uuid: "085c41158d3040fab5c0803a10952766"
format_stamp: "Formatted at 2026-09-23 17:27:30 on dist-test-slave-j0m4"
server_key: "4791348567cb8b8a50ef4b7e59a838eb"
server_key_iv: "28d2fedb047d1180fa9d26647b02c68c"
server_key_version: "encryptionkey@0"
I20260923 17:27:30.033459 19344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:27:30.037366 19359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:30.038728 19344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 17:27:30.038945 19344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/wal
uuid: "085c41158d3040fab5c0803a10952766"
format_stamp: "Formatted at 2026-09-23 17:27:30 on dist-test-slave-j0m4"
server_key: "4791348567cb8b8a50ef4b7e59a838eb"
server_key_iv: "28d2fedb047d1180fa9d26647b02c68c"
server_key_version: "encryptionkey@0"
I20260923 17:27:30.039224 19344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:30.058700 19344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:30.063407 19344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:30.063702 19344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:30.083392 19344 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.254:37297
I20260923 17:27:30.083416 19410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.254:37297 every 8 connection(s)
I20260923 17:27:30.084930 19344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:30.088896 19411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:30.094267 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 19344
I20260923 17:27:30.094504 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-0/wal/instance
I20260923 17:27:30.095000 28963 external_mini_cluster.cc:1428] Setting key 6dbb1eaf4de1a1a07ac56154738212c1
I20260923 17:27:30.096192 19411 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:27:30 dist-test-slave-j0m4 krb5kdc[19331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184450, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.72.253@KRBTEST.COM; defaulting to no policy
I20260923 17:27:30.122783 19414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.100886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.28.72.254:37297)
0923 17:27:30.101631 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:30.101642 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:30.101694 (+ 52us) server_negotiation.cc:407] Connection header received
0923 17:27:30.103070 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:30.103076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:30.103416 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:30.103605 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:30.105376 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.106584 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.108953 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.109549 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.109704 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:30.114529 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:30.114538 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:30.114553 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:30.114592 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:30.117889 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:30.118981 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:30.118988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:30.118990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:30.119240 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:30.119750 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:30.119753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:30.119756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:30.120317 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:30.120504 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:30.121282 (+ 778us) server_negotiation.cc:299] Negotiation successful
0923 17:27:30.121571 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":331,"threads_started":1}
I20260923 17:27:30.122828 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.099524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37297 (local address 127.0.0.1:34530)
0923 17:27:30.100398 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.100455 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:30.102606 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:30.103693 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:30.103704 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:30.104553 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:30.105075 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:30.105103 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.106762 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.106769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:30.108773 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:30.108795 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.109815 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.109824 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:30.110085 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:30.110866 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:30.110907 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:30.114185 (+ 3278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:30.118075 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:30.118093 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:30.118110 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:30.118798 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:30.119364 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:30.119371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:30.119375 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:30.119596 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:30.120519 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:30.120528 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:30.120793 (+ 265us) client_negotiation.cc:769] Sending connection context
0923 17:27:30.121120 (+ 327us) client_negotiation.cc:240] Negotiation successful
0923 17:27:30.121787 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":94,"thread_start_us":378,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.253@KRBTEST.COM" created.
I20260923 17:27:30.125914 19411 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:30.127048 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.126637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:34333 (local address 127.0.0.1:43622)
0923 17:27:30.126817 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.126937 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260923 17:27:30.127367 19363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
I20260923 17:27:30.129114 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.128648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:34333 (local address 127.0.0.1:43638)
0923 17:27:30.128828 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.128924 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 17:27:30.129367 19411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:34333: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.253.keytab.
Entry for principal kudu/127.28.72.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.253.keytab.
I20260923 17:27:30.158617 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.253:34333
--webserver_interface=127.28.72.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.253.keytab
--principal=kudu/127.28.72.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:37297,127.28.72.253:34333,127.28.72.252:43221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:27:30.178769 19411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } attempt: 1
I20260923 17:27:30.180100 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.179609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:34333 (local address 127.0.0.1:43642)
0923 17:27:30.179821 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.179953 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 17:27:30.181192 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.180806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:34333 (local address 127.0.0.1:43648)
0923 17:27:30.180956 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.181034 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:27:30.181381 19411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:34333: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
I20260923 17:27:30.249644 19411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } attempt: 2
I20260923 17:27:30.250782 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.250322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:34333 (local address 127.0.0.1:43654)
0923 17:27:30.250545 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.250648 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 17:27:30.251856 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.251454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:34333 (local address 127.0.0.1:43664)
0923 17:27:30.251657 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.251724 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:27:30.252038 19411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:34333: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
I20260923 17:27:30.343348 19411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } attempt: 3
I20260923 17:27:30.344578 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.344140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:34333 (local address 127.0.0.1:43666)
0923 17:27:30.344341 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.344444 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 17:27:30.345571 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.345209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:34333 (local address 127.0.0.1:43676)
0923 17:27:30.345350 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.345434 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:27:30.345753 19411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:34333: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
I20260923 17:27:30.489025 19411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } attempt: 4
I20260923 17:27:30.490316 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.489716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:34333 (local address 127.0.0.1:43688)
0923 17:27:30.490061 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.490160 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260923 17:27:30.491505 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.491026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:34333 (local address 127.0.0.1:43698)
0923 17:27:30.491222 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.491305 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 17:27:30.491714 19411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.253:34333: Network error: Client connection negotiation failed: client connection to 127.28.72.253:34333: connect: Connection refused (error 111)
W20260923 17:27:30.503086 19418 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:27:30.503597 19418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:30.503729 19418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:30.503880 19418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:30.511551 19418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:30.511689 19418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:30.511811 19418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:30.511898 19418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:30.512006 19418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:30.522429 19418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:37297,127.28.72.253:34333,127.28.72.252:43221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.253.keytab
--principal=kudu/127.28.72.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.253:34333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.72.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:30.523991 19418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:30.525367 19418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:30.537860 19423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:30.537832 19424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:30.538455 19426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:30.538990 19418 server_base.cc:1034] running on GCE node
I20260923 17:27:30.539925 19418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:30.541584 19418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:30.542908 19418 hybrid_clock.cc:648] HybridClock initialized: now 1790184450542819 us; error 61 us; skew 500 ppm
I20260923 17:27:30.543445 19418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:30 dist-test-slave-j0m4 krb5kdc[19331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184450, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:30.547091 19418 init.cc:377] Logged in from keytab as kudu/127.28.72.253@KRBTEST.COM (short username kudu)
I20260923 17:27:30.548775 19418 webserver.cc:460] Webserver started at http://127.28.72.253:39155/ using document root <none> and password file <none>
I20260923 17:27:30.549574 19418 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:30.549715 19418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:30.550069 19418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:30.553251 19418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/data/instance:
uuid: "adab58d8be2448d7aff46788a1b08a0d"
format_stamp: "Formatted at 2026-09-23 17:27:30 on dist-test-slave-j0m4"
server_key: "611f5ec61d19695a843dba72cde322a0"
server_key_iv: "be928106ac051f27387aced0009c6c61"
server_key_version: "encryptionkey@0"
I20260923 17:27:30.554133 19418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/wal/instance:
uuid: "adab58d8be2448d7aff46788a1b08a0d"
format_stamp: "Formatted at 2026-09-23 17:27:30 on dist-test-slave-j0m4"
server_key: "611f5ec61d19695a843dba72cde322a0"
server_key_iv: "be928106ac051f27387aced0009c6c61"
server_key_version: "encryptionkey@0"
I20260923 17:27:30.559860 19418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:27:30.563868 19433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:30.565388 19418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:27:30.565601 19418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/wal
uuid: "adab58d8be2448d7aff46788a1b08a0d"
format_stamp: "Formatted at 2026-09-23 17:27:30 on dist-test-slave-j0m4"
server_key: "611f5ec61d19695a843dba72cde322a0"
server_key_iv: "be928106ac051f27387aced0009c6c61"
server_key_version: "encryptionkey@0"
I20260923 17:27:30.565876 19418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:30.586421 19418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:30.591096 19418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:30.591426 19418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:30.611582 19418 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.253:34333
I20260923 17:27:30.611608 19484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.253:34333 every 8 connection(s)
I20260923 17:27:30.613039 19418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/data/info.pb
I20260923 17:27:30.616451 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 19418
I20260923 17:27:30.616645 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-1/wal/instance
I20260923 17:27:30.617122 28963 external_mini_cluster.cc:1428] Setting key 4b3574ec37334370ae179058e7c9088a
I20260923 17:27:30.617277 19485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:30.624156 19485 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:27:30 dist-test-slave-j0m4 krb5kdc[19331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184450, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.253@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.72.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.72.252@KRBTEST.COM" created.
I20260923 17:27:30.651197 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.627321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.28.72.254:37297)
0923 17:27:30.627906 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:30.627915 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:30.628967 (+ 1052us) server_negotiation.cc:407] Connection header received
0923 17:27:30.631949 (+ 2982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:30.631954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:30.632028 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:30.632130 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:30.634454 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.635346 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.637859 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.638341 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.638451 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:30.643204 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:30.643214 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:30.643222 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:30.643289 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:30.646499 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:30.647788 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:30.647798 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:30.647804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:30.647991 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:30.648536 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:30.648544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:30.648550 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:30.649034 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:30.649207 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:30.650211 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0923 17:27:30.650499 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":275,"threads_started":1}
I20260923 17:27:30.652182 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.627976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37297 (local address 127.0.0.1:34542)
0923 17:27:30.628779 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.628836 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:30.631622 (+ 2786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:30.632430 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:30.632443 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:30.633279 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:30.634130 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:30.634159 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.635526 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.635534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:30.637667 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:30.637687 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.638599 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.638608 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:30.638971 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:30.639901 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:30.639941 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:30.642874 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:30.646722 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:30.646732 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:30.646749 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:30.647584 (+ 835us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:30.648152 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:30.648156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:30.648158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:30.648385 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:30.649224 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:30.649236 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:30.649753 (+ 517us) client_negotiation.cc:769] Sending connection context
0923 17:27:30.650061 (+ 308us) client_negotiation.cc:240] Negotiation successful
0923 17:27:30.650619 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":303,"threads_started":1}
I20260923 17:27:30.654233 19485 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:27:30 dist-test-slave-j0m4 krb5kdc[19331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184450, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.253@KRBTEST.COM for kudu/127.28.72.253@KRBTEST.COM
I20260923 17:27:30.672606 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.655199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:34333 (local address 127.0.0.1:43702)
0923 17:27:30.655439 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.655460 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:30.655581 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:30.656533 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:30.656545 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:30.657096 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:30.657348 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:30.657363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.658925 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.658929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:30.659939 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:30.659951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.661355 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.661360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:30.661591 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:30.662341 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:30.662361 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:30.664799 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:30.669708 (+ 4909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:30.669713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:30.669716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:30.670197 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:30.670738 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:30.670744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:30.670750 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:30.670898 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:30.671585 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:30.671590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:30.671742 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 17:27:30.671885 (+ 143us) client_negotiation.cc:240] Negotiation successful
0923 17:27:30.672195 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 17:27:30.672694 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.655338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.28.72.253:34333)
0923 17:27:30.656071 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:30.656078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:30.656105 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:27:30.656156 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:30.656161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:30.656355 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:30.656482 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:30.657553 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.658762 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.660098 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.661164 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.661330 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:30.665094 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:30.665118 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:30.665141 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:30.665189 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:30.669509 (+ 4320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:30.670432 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:30.670447 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:30.670449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:30.670630 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:30.671013 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:30.671017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:30.671021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:30.671460 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:30.671685 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:30.672053 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 17:27:30.672322 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":575,"thread_start_us":429,"threads_started":1}
I20260923 17:27:30.675582 19485 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:30.676786 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.676317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:48992)
0923 17:27:30.676501 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.676623 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260923 17:27:30.677032 19434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
I20260923 17:27:30.678730 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.678336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49004)
0923 17:27:30.678530 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.678620 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 17:27:30.678930 19485 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43221: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.72.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.252.keytab.
Entry for principal kudu/127.28.72.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.252.keytab.
I20260923 17:27:30.687003 28963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
/tmp/dist-test-taskSv3JX5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.252:43221
--webserver_interface=127.28.72.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.252.keytab
--principal=kudu/127.28.72.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.72.212:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.72.254:37297,127.28.72.253:34333,127.28.72.252:43221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:27:30.728246 19485 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } attempt: 1
I20260923 17:27:30.729650 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.729053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49012)
0923 17:27:30.729310 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.729457 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260923 17:27:30.730792 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.730387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49024)
0923 17:27:30.730566 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.730662 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260923 17:27:30.731046 19485 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43221: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
I20260923 17:27:30.791006 19411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } attempt: 5
I20260923 17:27:30.799399 19485 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } attempt: 2
Sep 23 17:27:30 dist-test-slave-j0m4 krb5kdc[19331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184450, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for kudu/127.28.72.253@KRBTEST.COM
I20260923 17:27:30.800647 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.800083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49030)
0923 17:27:30.800339 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.800446 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260923 17:27:30.801898 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.801458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49040)
0923 17:27:30.801633 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.801723 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260923 17:27:30.802281 19485 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43221: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
I20260923 17:27:30.808084 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.791789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:34333 (local address 127.0.0.1:43704)
0923 17:27:30.792109 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.792138 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:30.792307 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:30.792796 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:30.792801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:30.793337 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:30.793570 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:30.793583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.794721 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.794726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:30.796147 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:30.796163 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.796955 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.796961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:30.797184 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:30.797941 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:30.797966 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:30.800868 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:30.805141 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:30.805145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:30.805148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:30.805773 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:30.806236 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:30.806240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:30.806242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:30.806388 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:30.807229 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:30.807233 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:30.807381 (+ 148us) client_negotiation.cc:769] Sending connection context
0923 17:27:30.807547 (+ 166us) client_negotiation.cc:240] Negotiation successful
0923 17:27:30.807783 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260923 17:27:30.808408 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.792007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.28.72.253:34333)
0923 17:27:30.792197 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:30.792202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:30.792222 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:27:30.792531 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:30.792536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:30.792620 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:30.792713 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:30.793773 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.794548 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.796322 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:30.796786 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:30.796886 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:30.801128 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:30.801134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:30.801137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:30.801172 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:30.804932 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:30.805926 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:30.805932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:30.805936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:30.806101 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:30.806523 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:30.806528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:30.806534 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:30.807018 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:30.807227 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:30.807735 (+ 508us) server_negotiation.cc:299] Negotiation successful
0923 17:27:30.808064 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:27:30.809526 19411 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:30.810636 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.810224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49048)
0923 17:27:30.810406 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.810495 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 17:27:30.811641 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.811299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49062)
0923 17:27:30.811446 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.811540 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:27:30.811846 19411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43221: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
I20260923 17:27:30.863131 19411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } attempt: 1
I20260923 17:27:30.864518 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.864025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49072)
0923 17:27:30.864237 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.864346 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 17:27:30.865610 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.865245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49074)
0923 17:27:30.865407 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.865487 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 17:27:30.865830 19411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43221: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
I20260923 17:27:30.893553 19485 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } attempt: 3
I20260923 17:27:30.895093 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.894309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49076)
0923 17:27:30.894631 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.894817 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260923 17:27:30.896623 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.895999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49078)
0923 17:27:30.896303 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.896428 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260923 17:27:30.896965 19485 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43221: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
I20260923 17:27:30.934048 19411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } attempt: 2
I20260923 17:27:30.935459 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.934870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49092)
0923 17:27:30.935171 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.935285 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 17:27:30.936461 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.936112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49104)
0923 17:27:30.936277 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.936346 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 17:27:30.936669 19411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43221: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
W20260923 17:27:31.038117 19493 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:27:31.038605 19493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:31.038733 19493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:31.038899 19493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 17:27:31.040215 19485 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } attempt: 4
I20260923 17:27:31.041646 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.041069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49106)
0923 17:27:31.041331 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:31.041443 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 17:27:31.042833 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.042420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49120)
0923 17:27:31.042590 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:31.042703 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 17:27:31.042912 19411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } attempt: 3
W20260923 17:27:31.043105 19485 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43221: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
I20260923 17:27:31.043918 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.043497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49128)
0923 17:27:31.043699 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:31.043776 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260923 17:27:31.044937 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.044554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49132)
0923 17:27:31.044749 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:31.044827 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260923 17:27:31.045169 19411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.72.252:43221: Network error: Client connection negotiation failed: client connection to 127.28.72.252:43221: connect: Connection refused (error 111)
W20260923 17:27:31.047690 19493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:31.047828 19493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:31.047945 19493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:31.048033 19493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:31.048143 19493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:31.058581 19493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.72.212:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.72.254:37297,127.28.72.253:34333,127.28.72.252:43221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/krb5kdc/kudu_127.28.72.252.keytab
--principal=kudu/127.28.72.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.72.252:43221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.72.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:27:31.060070 19493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:31.061444 19493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:31.074347 19498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:31.074245 19499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:31.074198 19501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:31.075548 19493 server_base.cc:1034] running on GCE node
I20260923 17:27:31.076433 19493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:31.077996 19493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:31.079259 19493 hybrid_clock.cc:648] HybridClock initialized: now 1790184451079185 us; error 47 us; skew 500 ppm
I20260923 17:27:31.079694 19493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:27:31 dist-test-slave-j0m4 krb5kdc[19331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184451, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:31.083279 19493 init.cc:377] Logged in from keytab as kudu/127.28.72.252@KRBTEST.COM (short username kudu)
I20260923 17:27:31.088517 19493 webserver.cc:460] Webserver started at http://127.28.72.252:33339/ using document root <none> and password file <none>
I20260923 17:27:31.089308 19493 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:31.089458 19493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:31.089795 19493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:31.092962 19493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/data/instance:
uuid: "61450b6db5204ae99468257e32616b10"
format_stamp: "Formatted at 2026-09-23 17:27:31 on dist-test-slave-j0m4"
server_key: "516463821e70d35677890028c949830c"
server_key_iv: "183263f9493642eb3fcb45a42504e542"
server_key_version: "encryptionkey@0"
I20260923 17:27:31.093909 19493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/wal/instance:
uuid: "61450b6db5204ae99468257e32616b10"
format_stamp: "Formatted at 2026-09-23 17:27:31 on dist-test-slave-j0m4"
server_key: "516463821e70d35677890028c949830c"
server_key_iv: "183263f9493642eb3fcb45a42504e542"
server_key_version: "encryptionkey@0"
I20260923 17:27:31.099761 19493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 17:27:31.103760 19508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:31.105357 19493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:27:31.105592 19493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/data,/tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/wal
uuid: "61450b6db5204ae99468257e32616b10"
format_stamp: "Formatted at 2026-09-23 17:27:31 on dist-test-slave-j0m4"
server_key: "516463821e70d35677890028c949830c"
server_key_iv: "183263f9493642eb3fcb45a42504e542"
server_key_version: "encryptionkey@0"
I20260923 17:27:31.105847 19493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:31.132938 19493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:31.137809 19493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:31.138118 19493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:31.157639 19493 rpc_server.cc:307] RPC server started. Bound to: 127.28.72.252:43221
I20260923 17:27:31.157681 19559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.72.252:43221 every 8 connection(s)
I20260923 17:27:31.159044 19493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/data/info.pb
I20260923 17:27:31.163033 19560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:31.165524 28963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu as pid 19493
I20260923 17:27:31.165709 28963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSv3JX5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184070511153-28963-0/minicluster-data/master-2/wal/instance
I20260923 17:27:31.166136 28963 external_mini_cluster.cc:1428] Setting key 7b4e49a8345af97c5da32a02e363a926
I20260923 17:27:31.171778 19560 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:27:31 dist-test-slave-j0m4 krb5kdc[19331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.252@KRBTEST.COM
Sep 23 17:27:31 dist-test-slave-j0m4 krb5kdc[19331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184451, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.252@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:27:31.189743 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.174807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.28.72.254:37297)
0923 17:27:31.175429 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.175436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.176092 (+ 656us) server_negotiation.cc:407] Connection header received
0923 17:27:31.176258 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.176264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.176366 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.176478 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.177614 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.178374 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.179810 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.180261 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.180427 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.184075 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:31.184081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:31.184085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:31.184121 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:31.186826 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.187663 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.187668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.187670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.187799 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.188197 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.188201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.188203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.188451 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:31.188619 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:31.189120 (+ 501us) server_negotiation.cc:299] Negotiation successful
0923 17:27:31.189346 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":339,"threads_started":1}
I20260923 17:27:31.190699 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.174814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.254:37297 (local address 127.0.0.1:34550)
0923 17:27:31.175925 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:31.175975 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:31.176122 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:31.176535 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:31.176540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:31.177177 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:31.177449 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:31.177460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.178537 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.178547 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:31.179648 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:31.179660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.180513 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.180520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:31.180682 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.181316 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:31.181354 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:31.183846 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:31.186989 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:31.187002 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:31.187009 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:31.187523 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:31.187913 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:31.187919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:31.187922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:31.188072 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:31.188582 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:31.188589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:31.188806 (+ 217us) client_negotiation.cc:769] Sending connection context
0923 17:27:31.188998 (+ 192us) client_negotiation.cc:240] Negotiation successful
0923 17:27:31.189402 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":692,"threads_started":1}
I20260923 17:27:31.190718 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.168245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49134 (local address 127.28.72.252:43221)
0923 17:27:31.169098 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.169113 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.169165 (+ 52us) server_negotiation.cc:407] Connection header received
0923 17:27:31.171594 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.171625 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.172202 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.172692 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.174180 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.175885 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.177460 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.178123 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.178328 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.182618 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:31.182627 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:31.182642 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:31.182677 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:31.186087 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.186854 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.186869 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.186877 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.187036 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.187480 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.187486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.187489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.188080 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:31.188415 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:31.188853 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 17:27:31.189591 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":605,"thread_start_us":391,"threads_started":1}
I20260923 17:27:31.192471 19560 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:27:31 dist-test-slave-j0m4 krb5kdc[19331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184451, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.252@KRBTEST.COM for kudu/127.28.72.253@KRBTEST.COM
Sep 23 17:27:31 dist-test-slave-j0m4 krb5kdc[19331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.254@KRBTEST.COM
I20260923 17:27:31.207090 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.193328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.28.72.253:34333)
0923 17:27:31.193552 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.193559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.193799 (+ 240us) server_negotiation.cc:407] Connection header received
0923 17:27:31.193946 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.193952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.194042 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.194154 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.195086 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.195861 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.197175 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.197606 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.197702 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.201101 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:31.201107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:31.201110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:31.201144 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:31.204150 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.205059 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.205064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.205066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.205204 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.205639 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.205642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.205644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.205937 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:31.206076 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:31.206614 (+ 538us) server_negotiation.cc:299] Negotiation successful
0923 17:27:31.206794 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 17:27:31.207209 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.193415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.253:34333 (local address 127.0.0.1:43714)
0923 17:27:31.193667 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:31.193706 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:31.193825 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:31.194177 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:31.194182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:31.194692 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:31.194900 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:31.194918 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.196010 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.196017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:31.197028 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:31.197043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.197753 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.197759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:31.197937 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.198523 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:31.198545 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:31.200898 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:31.204312 (+ 3414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:31.204318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:31.204322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:31.204911 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:31.205329 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:31.205334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:31.205338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:31.205517 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:31.206087 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:31.206093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:31.206296 (+ 203us) client_negotiation.cc:769] Sending connection context
0923 17:27:31.206469 (+ 173us) client_negotiation.cc:240] Negotiation successful
0923 17:27:31.206791 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260923 17:27:31.208293 19560 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:31.209735 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.193932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.28.72.254:37297)
0923 17:27:31.194101 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.194105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.194566 (+ 461us) server_negotiation.cc:407] Connection header received
0923 17:27:31.194730 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.194735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.194813 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.194902 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.196224 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.196946 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.198509 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.198899 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.198990 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.203032 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:31.203037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:31.203041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:31.203077 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:31.206442 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.207385 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.207391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.207394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.207570 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.208006 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.208010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.208013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.208289 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:31.208600 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:31.209210 (+ 610us) server_negotiation.cc:299] Negotiation successful
0923 17:27:31.209476 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 23 17:27:31 dist-test-slave-j0m4 krb5kdc[19331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184451, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.252@KRBTEST.COM for kudu/127.28.72.252@KRBTEST.COM
Sep 23 17:27:31 dist-test-slave-j0m4 krb5kdc[19331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.72.253@KRBTEST.COM
I20260923 17:27:31.222375 19411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } attempt: 4
I20260923 17:27:31.223753 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.209114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49142)
0923 17:27:31.209386 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:31.209400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:31.209540 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:31.209912 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:31.209917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:31.210413 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:31.210604 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:31.210614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.211714 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.211722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:31.213167 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:31.213186 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.213992 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.213996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:31.214174 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.214808 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:31.214825 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:31.217541 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:31.220526 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:31.220532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:31.220536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:31.220910 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:31.221341 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:31.221348 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:31.221351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:31.221493 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:31.222113 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:31.222118 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:31.222261 (+ 143us) client_negotiation.cc:769] Sending connection context
0923 17:27:31.222468 (+ 207us) client_negotiation.cc:240] Negotiation successful
0923 17:27:31.223205 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260923 17:27:31.224481 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.209098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.28.72.252:43221)
0923 17:27:31.209284 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.209291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.209493 (+ 202us) server_negotiation.cc:407] Connection header received
0923 17:27:31.209671 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.209676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.209745 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.209857 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.210755 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.211587 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.213371 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.213798 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.213968 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.217734 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:31.217738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:31.217741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:31.217778 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:31.220369 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.221028 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.221036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.221043 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.221223 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.221607 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.221612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.221615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.221936 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:31.222084 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:31.222553 (+ 469us) server_negotiation.cc:299] Negotiation successful
0923 17:27:31.224241 (+ 1688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1380,"server-negotiator.queue_time_us":67}
I20260923 17:27:31.230197 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.211823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.28.72.253:34333)
0923 17:27:31.211993 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.211997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.212037 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:27:31.212208 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.212211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.212267 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.212341 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.213937 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.214768 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.216741 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.217328 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.217441 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.222035 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:31.222041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:31.222046 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:31.222090 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:31.226775 (+ 4685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.227655 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.227660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.227662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.227809 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.228229 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.228234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.228236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.228506 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:31.228801 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:31.229684 (+ 883us) server_negotiation.cc:299] Negotiation successful
0923 17:27:31.229939 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:27:31.232205 19560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10: Bootstrap starting.
Sep 23 17:27:31 dist-test-slave-j0m4 krb5kdc[19331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184450, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.254@KRBTEST.COM for kudu/127.28.72.252@KRBTEST.COM
I20260923 17:27:31.235327 28963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:31.237771 19560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:31.239540 19560 log.cc:824] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:31.241273 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.223105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49144)
0923 17:27:31.223460 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:31.223501 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:31.223643 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:31.224762 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:31.224767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:31.225191 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:31.225467 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:31.225482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.227182 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.227191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:31.228813 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:31.228831 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.229846 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.229853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:31.230104 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.230802 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:31.230828 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:31.234565 (+ 3737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:31.238072 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:31.238078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:31.238082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:31.238629 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:31.239095 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:31.239100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:31.239104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:31.239371 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:31.240207 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:31.240212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:31.240378 (+ 166us) client_negotiation.cc:769] Sending connection context
0923 17:27:31.240526 (+ 148us) client_negotiation.cc:240] Negotiation successful
0923 17:27:31.240841 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260923 17:27:31.241297 19566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.223206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.28.72.252:43221)
0923 17:27:31.224417 (+ 1211us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.224421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.224438 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:27:31.224478 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.224481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.224565 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.225640 (+ 1075us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.225962 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.226981 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.228995 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.229670 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.229790 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.234826 (+ 5036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:31.234833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:31.234837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:31.234876 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:31.237913 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.238804 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.238810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.238813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.238974 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.239510 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.239515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.239561 (+ 46us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.240064 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:31.240227 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:31.240630 (+ 403us) server_negotiation.cc:299] Negotiation successful
0923 17:27:31.240837 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1113,"thread_start_us":891,"threads_started":1}
I20260923 17:27:31.246536 19411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766: Bootstrap starting.
I20260923 17:27:31.240141 19386 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "085c41158d3040fab5c0803a10952766"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34562
I20260923 17:27:31.250166 19560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10: No bootstrap required, opened a new log
I20260923 17:27:31.250182 19411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:31.251328 19411 log.cc:824] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:31.254367 19411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766: No bootstrap required, opened a new log
I20260923 17:27:31.259707 19411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } }
I20260923 17:27:31.260083 19411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085c41158d3040fab5c0803a10952766, State: Initialized, Role: FOLLOWER
I20260923 17:27:31.260550 19411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } }
I20260923 17:27:31.261895 19569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } } }
I20260923 17:27:31.262563 19411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:31.262645 19386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 17:27:31.262980 19386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:31.262763 19560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } }
I20260923 17:27:31.263346 19560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61450b6db5204ae99468257e32616b10, State: Initialized, Role: FOLLOWER
I20260923 17:27:31.262825 19569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:31.263903 19560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } }
I20260923 17:27:31.265462 19571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } } }
I20260923 17:27:31.266084 19571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:31.266824 19560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:31.267167 19386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } }
I20260923 17:27:31.271076 19386 leader_election.cc:288] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [CANDIDATE]: Term 1 election: Requested vote from peers adab58d8be2448d7aff46788a1b08a0d (127.28.72.253:34333), 61450b6db5204ae99468257e32616b10 (127.28.72.252:43221)
I20260923 17:27:31.270758 19535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "085c41158d3040fab5c0803a10952766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61450b6db5204ae99468257e32616b10"
I20260923 17:27:31.271646 19535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:31.276688 19535 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 085c41158d3040fab5c0803a10952766 in term 1.
I20260923 17:27:31.277662 19360 leader_election.cc:302] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085c41158d3040fab5c0803a10952766, 61450b6db5204ae99468257e32616b10; no voters:
I20260923 17:27:31.278374 19569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:31.290244 19569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [term 1 LEADER]: Becoming Leader. State: Replica: 085c41158d3040fab5c0803a10952766, State: Running, Role: LEADER
I20260923 17:27:31.291352 19569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } }
I20260923 17:27:31.306660 19585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 085c41158d3040fab5c0803a10952766. Latest consensus state: current_term: 1 leader_uuid: "085c41158d3040fab5c0803a10952766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } } }
I20260923 17:27:31.308339 19585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [sys.catalog]: This master's current role is: LEADER
W20260923 17:27:31.316825 19365 tablet.cc:2369] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:27:31.313536 19493 master_runner.cc:186] Error getting master registration for 127.28.72.254:37297: OK, instance_id { permanent_uuid: "085c41158d3040fab5c0803a10952766" instance_seqno: 1790184450077731 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:27:31.329043 19493 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:27:31.330339 19601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:31.342339 19485 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } attempt: 5
I20260923 17:27:31.346719 19566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.310593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.28.72.252:43221)
0923 17:27:31.310786 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.310792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.311713 (+ 921us) server_negotiation.cc:407] Connection header received
0923 17:27:31.312335 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.312341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.312455 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.312536 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.318456 (+ 5920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.320103 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.325652 (+ 5549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.326654 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.326801 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.335593 (+ 8792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:31.335604 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:31.335612 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:31.335674 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:31.340408 (+ 4734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.342018 (+ 1610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.342026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.342030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.342302 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.343170 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.343176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.343179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.343747 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:31.344326 (+ 579us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:31.345465 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0923 17:27:31.345977 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:27:31.347543 19603 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
Sep 23 17:27:31 dist-test-slave-j0m4 krb5kdc[19331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184450, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.72.253@KRBTEST.COM for kudu/127.28.72.252@KRBTEST.COM
W20260923 17:27:31.369004 19596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:27:31.369230 19596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:27:31.370198 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.343982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.28.72.252:43221)
0923 17:27:31.344212 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.344220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.344242 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:27:31.344303 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.344310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.344415 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.344530 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.345967 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.347709 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.350295 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.351141 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.351307 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.357261 (+ 5954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:31.357268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:31.357272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:31.357317 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:31.363637 (+ 6320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.365411 (+ 1774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.365422 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.365428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.365807 (+ 379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.366677 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.366682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.366716 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.367556 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:31.369167 (+ 1611us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:31.369537 (+ 370us) server_negotiation.cc:299] Negotiation successful
0923 17:27:31.369787 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 17:27:31.356755 19344 master_runner.cc:186] Error getting master registration for 127.28.72.253:34333: OK, instance_id { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" instance_seqno: 1790184450605974 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:27:31.370581 19344 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:27:31.370857 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.343069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.72.252:43221 (local address 127.0.0.1:49152)
0923 17:27:31.343555 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:31.343582 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:31.343720 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:31.344769 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:31.344777 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:31.345463 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:31.345745 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:31.345761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.348114 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.348123 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:31.350074 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:31.350091 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.351373 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.351381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:31.352431 (+ 1050us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.353237 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:31.353264 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:31.357028 (+ 3764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:31.364027 (+ 6999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:31.364035 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:31.364039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:31.365143 (+ 1104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:31.366095 (+ 952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:31.366102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:31.366107 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:31.366393 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:31.367778 (+ 1385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:31.367785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:31.368242 (+ 457us) client_negotiation.cc:769] Sending connection context
0923 17:27:31.368525 (+ 283us) client_negotiation.cc:240] Negotiation successful
0923 17:27:31.370416 (+ 1891us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301}
I20260923 17:27:31.370873 19601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:31.377279 19485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d: Bootstrap starting.
I20260923 17:27:31.380875 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.312736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34576 (local address 127.28.72.254:37297)
0923 17:27:31.312932 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.312940 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.333430 (+ 20490us) server_negotiation.cc:407] Connection header received
0923 17:27:31.333687 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.333694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.333799 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.333918 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.336485 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.338476 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.349468 (+ 10992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.350219 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.350340 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.373288 (+ 22948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:31.373294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:31.373299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:31.373345 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:31.377372 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.378346 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.378352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.378356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.378521 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.378966 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.378973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.378976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.379519 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:31.379844 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:31.380220 (+ 376us) server_negotiation.cc:299] Negotiation successful
0923 17:27:31.380490 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:27:31.382637 19485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:31.384140 19485 log.cc:824] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:31.388053 19485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d: No bootstrap required, opened a new log
I20260923 17:27:31.388478 19535 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10 [term 1 FOLLOWER]: Refusing update from remote peer 085c41158d3040fab5c0803a10952766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:27:31.389629 19585 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:27:31.390540 19514 tablet.cc:2369] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:27:31.393357 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.309982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.28.72.253:34333)
0923 17:27:31.310348 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.310354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.325180 (+ 14826us) server_negotiation.cc:407] Connection header received
0923 17:27:31.325633 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.325639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.325735 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.325811 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.334631 (+ 8820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.336027 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.347839 (+ 11812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.348598 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.348744 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.374063 (+ 25319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:31.374069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:31.374072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:31.374112 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:31.377967 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.381052 (+ 3085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.381061 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.381066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.381244 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:31.387342 (+ 6098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:31.387349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:31.387353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:31.387848 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:31.388177 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:31.392083 (+ 3906us) server_negotiation.cc:299] Negotiation successful
0923 17:27:31.392612 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260923 17:27:31.400600 19485 raft_consensus.cc:348] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } }
I20260923 17:27:31.401141 19485 raft_consensus.cc:729] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adab58d8be2448d7aff46788a1b08a0d, State: Initialized, Role: FOLLOWER
I20260923 17:27:31.401762 19485 consensus_queue.cc:260] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } }
I20260923 17:27:31.405366 19571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 085c41158d3040fab5c0803a10952766. Latest consensus state: current_term: 1 leader_uuid: "085c41158d3040fab5c0803a10952766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } } }
I20260923 17:27:31.405781 19571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:31.410501 19485 sys_catalog.cc:564] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:31.411015 19459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:31.412990 19585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "085c41158d3040fab5c0803a10952766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } } }
I20260923 17:27:31.413290 19585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:31.415295 19460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "085c41158d3040fab5c0803a10952766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "adab58d8be2448d7aff46788a1b08a0d"
I20260923 17:27:31.416055 19459 raft_consensus.cc:1261] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d [term 1 FOLLOWER]: Refusing update from remote peer 085c41158d3040fab5c0803a10952766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:27:31.416234 19571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "085c41158d3040fab5c0803a10952766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } } }
I20260923 17:27:31.416509 19571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:31.417910 19601 catalog_manager.cc:1347] Generated new cluster ID: 5cf40c5562614373b4b4e829bdd65252
I20260923 17:27:31.418046 19601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:31.417914 19605 mvcc.cc:205] Tried to move back new op lower bound from 7332595512873852928 to 7332595512498487296. Current Snapshot: MvccSnapshot[applied={T|T < 7332595512873852928 or (T in {7332595512873852928})}]
I20260923 17:27:31.419957 19460 raft_consensus.cc:2454] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 085c41158d3040fab5c0803a10952766 in term 1.
I20260923 17:27:31.421005 19609 sys_catalog.cc:455] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } } }
I20260923 17:27:31.421381 19609 sys_catalog.cc:458] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:31.421388 19607 mvcc.cc:205] Tried to move back new op lower bound from 7332595512873852928 to 7332595512498487296. Current Snapshot: MvccSnapshot[applied={T|T < 7332595512873852928 or (T in {7332595512873852928})}]
I20260923 17:27:31.421865 19569 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:27:31.452809 19601 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:31.455471 19601 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:31.482870 19601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766: Generated new TSK 0
I20260923 17:27:31.484651 19601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:31.487749 19609 sys_catalog.cc:455] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 085c41158d3040fab5c0803a10952766. Latest consensus state: current_term: 1 leader_uuid: "085c41158d3040fab5c0803a10952766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } } }
I20260923 17:27:31.488189 19609 sys_catalog.cc:458] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:31.487720 19624 sys_catalog.cc:455] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "085c41158d3040fab5c0803a10952766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } } }
I20260923 17:27:31.488399 19624 sys_catalog.cc:458] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:31.489944 19569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "085c41158d3040fab5c0803a10952766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085c41158d3040fab5c0803a10952766" member_type: VOTER last_known_addr { host: "127.28.72.254" port: 37297 } } peers { permanent_uuid: "adab58d8be2448d7aff46788a1b08a0d" member_type: VOTER last_known_addr { host: "127.28.72.253" port: 34333 } } peers { permanent_uuid: "61450b6db5204ae99468257e32616b10" member_type: VOTER last_known_addr { host: "127.28.72.252" port: 43221 } } }
I20260923 17:27:31.490355 19569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085c41158d3040fab5c0803a10952766 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:31.535722 19626 catalog_manager.cc:1259] Loaded cluster ID: 5cf40c5562614373b4b4e829bdd65252
I20260923 17:27:31.535882 19626 catalog_manager.cc:1552] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d: loading cluster ID for follower catalog manager: success
I20260923 17:27:31.540675 19626 catalog_manager.cc:1574] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d: acquiring CA information for follower catalog manager: success
I20260923 17:27:31.544000 19626 catalog_manager.cc:1602] T 00000000000000000000000000000000 P adab58d8be2448d7aff46788a1b08a0d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 17:27:31.641096 19564 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:31.628546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.28.72.254:37297)
0923 17:27:31.628783 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.628790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.637709 (+ 8919us) server_negotiation.cc:407] Connection header received
0923 17:27:31.638855 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.638862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.638958 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.639036 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.640840 (+ 1804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34588: BlockingRecv error: recv got EOF from 127.0.0.1:34588 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260923 17:27:31.646185 19490 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:31.632491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.28.72.253:34333)
0923 17:27:31.632742 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.632751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.643259 (+ 10508us) server_negotiation.cc:407] Connection header received
0923 17:27:31.643974 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.643984 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.644178 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.644279 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.645821 (+ 1542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43730: BlockingRecv error: recv got EOF from 127.0.0.1:43730 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260923 17:27:31.648981 19563 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:31.630702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.28.72.252:43221)
0923 17:27:31.630969 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.630978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.643833 (+ 12855us) server_negotiation.cc:407] Connection header received
0923 17:27:31.647019 (+ 3186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.647029 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.647217 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.647333 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.648726 (+ 1393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49156: BlockingRecv error: recv got EOF from 127.0.0.1:49156 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
I20260923 17:27:31.701776 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.685609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.28.72.253:34333)
0923 17:27:31.685838 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.685845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.692873 (+ 7028us) server_negotiation.cc:407] Connection header received
0923 17:27:31.693498 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.693505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.693741 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.693910 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:31.694687 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.695989 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.698392 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.699166 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.699281 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.700132 (+ 851us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:31.700877 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:31.701012 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:31.701273 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 17:27:31.701393 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260923 17:27:31.752426 19563 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:31.738474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.28.72.252:43221)
0923 17:27:31.738674 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.738681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.750030 (+ 11349us) server_negotiation.cc:407] Connection header received
0923 17:27:31.750692 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.750699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.750841 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.750927 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.752135 (+ 1208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49164: BlockingRecv error: recv got EOF from 127.0.0.1:49164 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:27:31.803833 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.788960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43746 (local address 127.28.72.253:34333)
0923 17:27:31.789152 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.789158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.796144 (+ 6986us) server_negotiation.cc:407] Connection header received
0923 17:27:31.796737 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.796743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.796843 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.796912 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:31.797701 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.798897 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.801020 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.801781 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.801889 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.802593 (+ 704us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:31.802794 (+ 201us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:31.802993 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:31.803311 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 17:27:31.803421 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260923 17:27:31.858270 19563 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:31.846091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49170 (local address 127.28.72.252:43221)
0923 17:27:31.846331 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.846338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.855805 (+ 9467us) server_negotiation.cc:407] Connection header received
0923 17:27:31.856191 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.856200 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.856336 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.856427 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.858005 (+ 1578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49170: BlockingRecv error: recv got EOF from 127.0.0.1:49170 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:27:31.934341 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.906630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.28.72.253:34333)
0923 17:27:31.906877 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.906886 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.916203 (+ 9317us) server_negotiation.cc:407] Connection header received
0923 17:27:31.916591 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.916598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.916713 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.916852 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:31.927979 (+ 11127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.929448 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.931778 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:31.932578 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:31.932767 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:31.933356 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:31.933536 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:31.933645 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:31.933799 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 17:27:31.933901 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260923 17:27:31.974507 19563 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:31.972381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49184 (local address 127.28.72.252:43221)
0923 17:27:31.972797 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:31.972805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:31.972843 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:27:31.972930 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:31.972937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:31.973048 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:31.973165 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:31.974203 (+ 1038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49184: BlockingRecv error: recv got EOF from 127.0.0.1:49184 (error 108)
Metrics: {"server-negotiator.queue_time_us":274}
I20260923 17:27:32.043188 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:32.031015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43768 (local address 127.28.72.253:34333)
0923 17:27:32.031260 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.031267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.035871 (+ 4604us) server_negotiation.cc:407] Connection header received
0923 17:27:32.036336 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.036344 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.036451 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.036524 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:32.037275 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.038507 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.040704 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.041427 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.041539 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:32.042267 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:32.042431 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:32.042512 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:32.042749 (+ 237us) server_negotiation.cc:299] Negotiation successful
0923 17:27:32.042838 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260923 17:27:32.117489 19563 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:32.097476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49190 (local address 127.28.72.252:43221)
0923 17:27:32.097757 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.097763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.115515 (+ 17752us) server_negotiation.cc:407] Connection header received
0923 17:27:32.115856 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.115862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.115967 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.116052 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:32.117204 (+ 1152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49190: BlockingRecv error: recv got EOF from 127.0.0.1:49190 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
I20260923 17:27:32.159250 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:32.152068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.28.72.253:34333)
0923 17:27:32.152261 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.152267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.152854 (+ 587us) server_negotiation.cc:407] Connection header received
0923 17:27:32.153051 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.153058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.153158 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.153272 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:32.153793 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.155136 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.156833 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.157588 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.157714 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:32.158298 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:32.158456 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:32.158531 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:32.158778 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 17:27:32.158881 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 17:27:32.179338 19563 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:32.175456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.28.72.252:43221)
0923 17:27:32.175653 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.175659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.177635 (+ 1976us) server_negotiation.cc:407] Connection header received
0923 17:27:32.177834 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.177840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.177955 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.178090 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:32.179081 (+ 991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49196: BlockingRecv error: recv got EOF from 127.0.0.1:49196 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:27:32.205173 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:32.199068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.28.72.253:34333)
0923 17:27:32.199302 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.199306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.199731 (+ 425us) server_negotiation.cc:407] Connection header received
0923 17:27:32.199906 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.199910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.200014 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.200109 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:32.200700 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.201663 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.203213 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.203705 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.203815 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:32.204390 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:32.204526 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:32.204602 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:32.204834 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 17:27:32.204941 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260923 17:27:32.214723 19563 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:32.212662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.28.72.252:43221)
0923 17:27:32.212902 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.212906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.213327 (+ 421us) server_negotiation.cc:407] Connection header received
0923 17:27:32.213547 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.213552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.213635 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.213727 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:32.214551 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49210: BlockingRecv error: recv got EOF from 127.0.0.1:49210 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
I20260923 17:27:32.256331 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:32.250858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.28.72.253:34333)
0923 17:27:32.251225 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.251232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.251523 (+ 291us) server_negotiation.cc:407] Connection header received
0923 17:27:32.251760 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.251765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.251856 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.251967 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:32.252430 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.253145 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.254543 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.255000 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.255175 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:32.255561 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:32.255678 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:32.255778 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:32.255938 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 17:27:32.256031 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
W20260923 17:27:32.264637 19563 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:32.262437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.28.72.252:43221)
0923 17:27:32.262693 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.262700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.262958 (+ 258us) server_negotiation.cc:407] Connection header received
0923 17:27:32.263210 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.263215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.263291 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.263391 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:32.264324 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49218: BlockingRecv error: recv got EOF from 127.0.0.1:49218 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 17:27:32.338402 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:32.332492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.28.72.253:34333)
0923 17:27:32.332802 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.332809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.333007 (+ 198us) server_negotiation.cc:407] Connection header received
0923 17:27:32.333265 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.333270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.333352 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.333460 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:32.333876 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.334997 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.336579 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.337020 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.337215 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:32.337642 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:32.337772 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:32.337873 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:32.338026 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 17:27:32.338109 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260923 17:27:32.346801 19563 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:32.344628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.28.72.252:43221)
0923 17:27:32.344912 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.344921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.345216 (+ 295us) server_negotiation.cc:407] Connection header received
0923 17:27:32.345529 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.345534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.345624 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.345740 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:32.346556 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49230: BlockingRecv error: recv got EOF from 127.0.0.1:49230 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260923 17:27:32.371411 19596 catalog_manager.cc:1259] Loaded cluster ID: 5cf40c5562614373b4b4e829bdd65252
I20260923 17:27:32.371547 19596 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10: loading cluster ID for follower catalog manager: success
I20260923 17:27:32.374089 19596 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10: acquiring CA information for follower catalog manager: success
I20260923 17:27:32.375998 19596 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 61450b6db5204ae99468257e32616b10: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:27:32.485395 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:32.479593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.28.72.253:34333)
0923 17:27:32.479860 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.479865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.480203 (+ 338us) server_negotiation.cc:407] Connection header received
0923 17:27:32.480403 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.480414 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.480491 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.480585 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:32.481071 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.481951 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.483559 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.484020 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.484133 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:32.484672 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:32.484785 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:32.484873 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:32.485042 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 17:27:32.485117 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 17:27:32.498243 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:32.491959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.28.72.252:43221)
0923 17:27:32.492205 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.492212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.492611 (+ 399us) server_negotiation.cc:407] Connection header received
0923 17:27:32.492774 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.492778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.492945 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.493046 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:32.493507 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.494200 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.495824 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.496233 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.496336 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:32.496976 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:32.497575 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:32.497698 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:32.497819 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 17:27:32.497895 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:27:32.521526 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:32.505288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.28.72.252:43221)
0923 17:27:32.505638 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.505648 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.514671 (+ 9023us) server_negotiation.cc:407] Connection header received
0923 17:27:32.514919 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.514930 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.515080 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.515253 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:32.516111 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.517062 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.519162 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.519942 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.520101 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:32.520676 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:32.520826 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:32.520922 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:32.521097 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 17:27:32.521193 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260923 17:27:32.522468 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:32.505141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.28.72.254:37297)
0923 17:27:32.515259 (+ 10118us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.515267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.515290 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:27:32.515430 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.515437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.515682 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.515943 (+ 261us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:32.516456 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.517545 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.519886 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.520317 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.520457 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:32.521037 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:32.521523 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:32.521913 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:32.521991 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 17:27:32.522101 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9948,"thread_start_us":9841,"threads_started":1}
I20260923 17:27:32.522559 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:32.505920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.28.72.253:34333)
0923 17:27:32.506149 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:32.506156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:32.515688 (+ 9532us) server_negotiation.cc:407] Connection header received
0923 17:27:32.515861 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:32.515868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:32.515975 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:32.516076 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:32.517002 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.518023 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.520217 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:32.520896 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:32.521035 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:32.521481 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:32.521642 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:32.521737 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:32.522175 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 17:27:32.522268 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:27:32.528714 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 19344
I20260923 17:27:32.529940 19406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:27:32.642827 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 19344
I20260923 17:27:32.665198 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 19418
I20260923 17:27:32.666553 19480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:27:32.777076 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 19418
I20260923 17:27:32.798659 28963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu pid 19493
I20260923 17:27:32.800036 19555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:27:32.909148 28963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSv3JX5/build/asan/bin/kudu with pid 19493
2026-09-23T17:27:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4698 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9676 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (382345 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST